drawings of a musical.ly user using hand signs

You know how we add emoji to texts?  In a face-to-face conversation, we don’t communicate simply with words, we also use facial expressions, tone of voice, gestures and body language, and sometimes touch. Emojis are pictograms that let us express some of these things in a textual medium. I think that as social media are becoming more video-based, we’re going to be seeing new kinds of pictograms that do the same work as emoji do in text, but that will work for video.

I wrote a paper about this that was just published in Social Media and Society, which is an open access journal that has published some really fabulous papers in social media and internet studies. It’s called Hand Signs for Lip-syncing: The Emergence of a Gestural Language on Musical.ly as a Video-Based Equivalent to Emoji. As you might have guessed, it argues that the hand signs lip-syncs on musical.ly use are doing what emoji do for text – but in video.

Musical.ly is super popular with tweens and teens, but for those of you not in the know, here is an example of how the hand signs work on musical.ly.

Musical.ly has become a pretty diverse video-sharing app, but it started as a lip-syncing app, and lip-syncing is still a major part of musical.ly. You record 15 second videos of yourself singing to a tune that you picked from the app’s library. You can add filters and special effects, but you can’t add text or your own voice.

I think the fact that the modalities are limited – you can have video but no voice or text – leads to the development of a pictogram to make up for that limitation. That’s exactly what happened with text-based communication. Emoticons came early, and were standardised as emoji 🙂 after a while.

Hand signs on musical.ly are pretty well defined. Looking at the videos or the tutorials on YouTube you’ll see that there are many that are quite standard. They’re usually made with just one hand, since the camera is held in the other hand, and often camera movements are important too, but more as a dance beat than as a unit of meaning. Here are the hand signs used by one lip-syncer to perform a 15 second sample from the song “Too Good” by Drake and Rihanna. First, she performs the words “I’m way too good to you,” using individual signs for “too”, “good”, “to” and “you”.

drawings of a musical.ly user using hand signs

The next words are “You take my love for granted/I just don’t understand it.” This is harder to translate into signs word for word, so the lip-syncer interprets it in just three signs, pointing to indicate “you”, shaping her fingers into half of a heart for “love”, and pointing to her head for “understand”.

drawings of a musical.ly user using hand signs

Looking at a lot of tutorials on YouTube (I love Nigeria Blessings’ tutorial) and at a lot of individual lip-syncing videos, I came up with a very incomplete list of some common signs used on musical.ly:

In my paper I talk about how these hand signs are similar to the codified gestures used in early oratory and in theatre. These are called chironomics, and there are 17th and 18th century books explaining them in detail. The drawings are fascinating:

I think it’s important to think of the hand signs as performance, and in the theatrical or musical sense, not in the more generalised sense that Goffman used for a metaphor, where all social interaction is “performative”. No, these are literal performances, interpretations of a script for an audience. That’s important, because without realising that, we might think the hand signs are just redundant. After all, they’re just repeating the same things that are said in the lyrics of the song, but using signs. When we think of the signs as part of a performance, though, we realise that they’re an interpretation, not simply a repetition. Each muser uses hand signs slightly differently.

And those hand signs aren’t easy. Just look at Baby Ariel, who is very popular on musical.ly, trying to teach her mother to  lip-sync. Or look at me in my Snapchat Research story trying to explain hand gestures on musical.ly just as I was starting to write the paper that was published this week:

The full paper, which is finally published after two rounds of Revise & Resubmit (it’s way better now) is open access, so free to read for anyone.

Oh, and sweethearts, if you feel like tweeting a link to the paper, it ups my Altmetrics. That makes the paper more visible. How about we all tweet each others papers and we’ll all be famous? ?

Leave A Comment

Recommended Posts

Machine Vision

Cultural Representations of Machine Vision: An Experimental Mixed Methods Workshop

Call for submissions to a workshop, Bergen, Norway
Workshop dates: 15-17 August 2022
Proposals due: 15 June

The Machine Vision in Everyday Life project invites proposals for an interdisciplinary workshop using qualitative approaches and digital methods to analyse how machine vision is represented in art, science fiction, games, social media and other forms of cultural and aesthetic expression.

Digital Humanities Machine Vision

What do different machine vision technologies do in fiction and art?

For the Machine Vision in Everyday Life project we’ve analysed how machine vision technologies are portrayed and used in 500 works of fiction and art, including 77 digital games, 190 digital artworks and 233 movies, novels and other narratives. You can browse […]

AI and algorithmic culture Presentations

My talk on caring AIs in recent sci-fi novels

I’m giving a talk at an actual f2f academic conference today, Critical Borders, Radical Re(visions) of AI, in Cambridge. I was particularly excited to see this conference because it’s organised by the people who edited AI Narratives A History of Imaginative Thinking […]