You know how we add emoji to texts?  In a face-to-face conversation, we don’t communicate simply with words, we also use facial expressions, tone of voice, gestures and body language, and sometimes touch. Emojis are pictograms that let us express some of these things in a textual medium. I think that as social media are becoming more video-based, we’re going to be seeing new kinds of pictograms that do the same work as emoji do in text, but that will work for video.

I wrote a paper about this that was just published in Social Media and Society, which is an open access journal that has published some really fabulous papers in social media and internet studies. It’s called Hand Signs for Lip-syncing: The Emergence of a Gestural Language on Musical.ly as a Video-Based Equivalent to Emoji. As you might have guessed, it argues that the hand signs lip-syncs on musical.ly use are doing what emoji do for text – but in video.

Musical.ly is super popular with tweens and teens, but for those of you not in the know, here is an example of how the hand signs work on musical.ly.

Musical.ly has become a pretty diverse video-sharing app, but it started as a lip-syncing app, and lip-syncing is still a major part of musical.ly. You record 15 second videos of yourself singing to a tune that you picked from the app’s library. You can add filters and special effects, but you can’t add text or your own voice.

I think the fact that the modalities are limited – you can have video but no voice or text – leads to the development of a pictogram to make up for that limitation. That’s exactly what happened with text-based communication. Emoticons came early, and were standardised as emoji 🙂 after a while.

Hand signs on musical.ly are pretty well defined. Looking at the videos or the tutorials on YouTube you’ll see that there are many that are quite standard. They’re usually made with just one hand, since the camera is held in the other hand, and often camera movements are important too, but more as a dance beat than as a unit of meaning. Here are the hand signs used by one lip-syncer to perform a 15 second sample from the song “Too Good” by Drake and Rihanna. First, she performs the words “I’m way too good to you,” using individual signs for “too”, “good”, “to” and “you”.

The next words are “You take my love for granted/I just don’t understand it.” This is harder to translate into signs word for word, so the lip-syncer interprets it in just three signs, pointing to indicate “you”, shaping her fingers into half of a heart for “love”, and pointing to her head for “understand”.

Looking at a lot of tutorials on YouTube (I love Nigeria Blessings’ tutorial) and at a lot of individual lip-syncing videos, I came up with a very incomplete list of some common signs used on musical.ly:

In my paper I talk about how these hand signs are similar to the codified gestures used in early oratory and in theatre. These are called chironomics, and there are 17th and 18th century books explaining them in detail. The drawings are fascinating:

I think it’s important to think of the hand signs as performance, and in the theatrical or musical sense, not in the more generalised sense that Goffman used for a metaphor, where all social interaction is “performative”. No, these are literal performances, interpretations of a script for an audience. That’s important, because without realising that, we might think the hand signs are just redundant. After all, they’re just repeating the same things that are said in the lyrics of the song, but using signs. When we think of the signs as part of a performance, though, we realise that they’re an interpretation, not simply a repetition. Each muser uses hand signs slightly differently.

And those hand signs aren’t easy. Just look at Baby Ariel, who is very popular on musical.ly, trying to teach her mother to  lip-sync. Or look at me in my Snapchat Research story trying to explain hand gestures on musical.ly just as I was starting to write the paper that was published this week:

The full paper, which is finally published after two rounds of Revise & Resubmit (it’s way better now) is open access, so free to read for anyone.

Oh, and sweethearts, if you feel like tweeting a link to the paper, it ups my Altmetrics. That makes the paper more visible. How about we all tweet each others papers and we’ll all be famous? ?