drawings of a musical.ly user using hand signs

You know how we add emoji to texts?  In a face-to-face conversation, we don’t communicate simply with words, we also use facial expressions, tone of voice, gestures and body language, and sometimes touch. Emojis are pictograms that let us express some of these things in a textual medium. I think that as social media are becoming more video-based, we’re going to be seeing new kinds of pictograms that do the same work as emoji do in text, but that will work for video.

I wrote a paper about this that was just published in Social Media and Society, which is an open access journal that has published some really fabulous papers in social media and internet studies. It’s called Hand Signs for Lip-syncing: The Emergence of a Gestural Language on Musical.ly as a Video-Based Equivalent to Emoji. As you might have guessed, it argues that the hand signs lip-syncs on musical.ly use are doing what emoji do for text – but in video.

Musical.ly is super popular with tweens and teens, but for those of you not in the know, here is an example of how the hand signs work on musical.ly.

Musical.ly has become a pretty diverse video-sharing app, but it started as a lip-syncing app, and lip-syncing is still a major part of musical.ly. You record 15 second videos of yourself singing to a tune that you picked from the app’s library. You can add filters and special effects, but you can’t add text or your own voice.

I think the fact that the modalities are limited – you can have video but no voice or text – leads to the development of a pictogram to make up for that limitation. That’s exactly what happened with text-based communication. Emoticons came early, and were standardised as emoji 🙂 after a while.

Hand signs on musical.ly are pretty well defined. Looking at the videos or the tutorials on YouTube you’ll see that there are many that are quite standard. They’re usually made with just one hand, since the camera is held in the other hand, and often camera movements are important too, but more as a dance beat than as a unit of meaning. Here are the hand signs used by one lip-syncer to perform a 15 second sample from the song “Too Good” by Drake and Rihanna. First, she performs the words “I’m way too good to you,” using individual signs for “too”, “good”, “to” and “you”.

drawings of a musical.ly user using hand signs

The next words are “You take my love for granted/I just don’t understand it.” This is harder to translate into signs word for word, so the lip-syncer interprets it in just three signs, pointing to indicate “you”, shaping her fingers into half of a heart for “love”, and pointing to her head for “understand”.

drawings of a musical.ly user using hand signs

Looking at a lot of tutorials on YouTube (I love Nigeria Blessings’ tutorial) and at a lot of individual lip-syncing videos, I came up with a very incomplete list of some common signs used on musical.ly:

In my paper I talk about how these hand signs are similar to the codified gestures used in early oratory and in theatre. These are called chironomics, and there are 17th and 18th century books explaining them in detail. The drawings are fascinating:

I think it’s important to think of the hand signs as performance, and in the theatrical or musical sense, not in the more generalised sense that Goffman used for a metaphor, where all social interaction is “performative”. No, these are literal performances, interpretations of a script for an audience. That’s important, because without realising that, we might think the hand signs are just redundant. After all, they’re just repeating the same things that are said in the lyrics of the song, but using signs. When we think of the signs as part of a performance, though, we realise that they’re an interpretation, not simply a repetition. Each muser uses hand signs slightly differently.

And those hand signs aren’t easy. Just look at Baby Ariel, who is very popular on musical.ly, trying to teach her mother to  lip-sync. Or look at me in my Snapchat Research story trying to explain hand gestures on musical.ly just as I was starting to write the paper that was published this week:

The full paper, which is finally published after two rounds of Revise & Resubmit (it’s way better now) is open access, so free to read for anyone.

Oh, and sweethearts, if you feel like tweeting a link to the paper, it ups my Altmetrics. That makes the paper more visible. How about we all tweet each others papers and we’ll all be famous? ?

Leave A Comment

Recommended Posts

Triple book talk: Watch James Dobson, Jussi Parikka and me discuss our 2023 books

Thanks to everyone who came to the triple book talk of three recent books on machine vision by James Dobson, Jussi Parikka and me, and thanks for excellent questions. Several people have emailed to asked if we recorded it, and yes we did! Here you go! James and Jussi’s books […]

Image on a black background of a human hand holding a graphic showing the word AI with a blue circuit board pattern inside surrounded by blurred blue and yellow dots and a concentric circular blue design.
AI and algorithmic culture Machine Vision

Four visual registers for imaginaries of machine vision

I’m thrilled to announce another publication from our European Research Council (ERC)-funded research project on Machine Vision: Gabriele de Setaand Anya Shchetvina‘s paper analysing how Chinese AI companies visually present machine vision technologies. They find that the Chinese machine vision imaginary is global, blue and competitive.  De Seta, Gabriele, and Anya Shchetvina. “Imagining Machine […]

Do people flock to talks about ChatGPT because they are scared?

Whenever I give talks about ChatGPT and LLMs, whether to ninth graders, businesses or journalists, I meet people who are hungry for information, who really want to understand this new technology. I’ve interpreted this as interest and a need to understand – but yesterday, Eirik Solheim said that every time […]