I gave a talk at the Moral Machines symposium in Helsinki last year, and just heard that a revised version of the talk will be published in an anthology tentatively titled The Ethos of Digital Environments: Technology, Literary Theory and Philosophy. The anthology is edited by Hanna-Riikka Roine and Susanna Lindberg and will be published by Routledge, presumably in 2021 or 2022. Here is an excerpt from my draft of the chapter, where I explore the idea that there might not be that much difference between a neural network that can predict when a human would cry and that involuntary tightness we humans sometimes feel in our chests when we watch a sad movie.

Emotions are often conceived as the determining difference between humans and machines, and indeed, between groups of humans and whatever or whoever they wish to define as non-human. “They don’t have the same feelings we do,” the narrator imagines the wives thinking of the handmaids in Margaret Atwood’s novel (1986, 215); “they don’t seem to feel anything, no pleasure, no pain”, the Terrans remark of the indigenous people they rape and beat in Ursula le Guin’s The Word for World is Forest (1972, 18).

“No text-analysis program weeps when it reads the passages in Felix Salten’s Bambi in which Bambi’s mother dies”, Drucker wrote in “What Distant Reading Isn’t”, as though this ability to weep, to respond emotionally and not just analytically is our defining difference. A neural network would not weep, it is true, but it could certainly learn to identify texts that would typically make a human weep. How different is that from actually feeling?

 If machines can feel, does that make them human? Should they have rights as people do? In Bladerunner, replicants are subjected to a test to make sure they do not have human emotions, and are allowed no more than a four-year lifespan so they do not develop emotions. Once machines feel, humans can also develop feelings for them, as in the many stories where humans fall in love with robots, as Drecker falls in love with the replicant Rachael in Bladerunner. 

One of the ethical arguments against killer robots, or lethal autonomous weapon systems (LAWS), is that they kill without emotion, as Daniel Lim describes in a paper discussing the different ethical arguments used against LAWS:

LAWS do not have a capacity to have a notion of sacrifice or a capacity to fathom the significance of using force against a human because they don’t feel anything—they have no emotions. It is only because humans can feel the rage and agony that accompanies the killing of humans that they can understand sacrifice and the use of force against a human. Only then can they realize the ‘gravity of the decision’ to kill. (Lim 2019)

Lim disagrees with this argument. Not all people feel the appropriate emotions when killing in battle either, he points out. While many humans may cry when they read of Bambi’s mother’s death, not all humans will. Can we really separate human from machine based on emotion? We have arrived at a moral impasse. 

Another approach is to use N. Katherine Hayles’s understanding of cognition as something that is common to humans, animals and many technical systems. Hayles defines cognition as “a process that interprets information within contexts that connect it with meaning” (Hayles 2017, 22). This definition covers what a car with a collision avoidance system does when it gathers data from its surroundings and interprets it in order to determine whether a dangerous situation is occurring, upon which it overrides the human driver and takes evasive measures. It also covers a human’s non-conscious cognition as a human body gathers data about its surroundings and adjusts bodily functions accordingly. For instance, when I chop an onion, my eyes sting as the onion releases a chemical irritant into the air, and in response, my eyes’ lachrymal glands release tears. But this has nothing to do with emotions, you might object, it is simply biomechanics. And humans can cry without being sad. For their artwork Tear Dealer, Polish artists Alicja Rogalska and Lukasz Surowiec set up a temporary shop where people could “produce and sell their tears for approximately €25 per 3ml” (2014). The video installation documenting the process shows redfaced people listening to presumably sad music or a sad story through headphones, weeping, some wailing, apparently genuinely wracked with emotion. Each person then clinically lifts a test tube to the corner of each of their eyes to capture tears. One woman laughs to a friend when comparing how much liquid each of them has captured. 

Think of a time when you felt a strong urge to cry and you tried to stop yourself. Were the emotions and their symptom (the constrained chest and throat, the reddening eyes, the hormones flowing through your veins) consciously, rationally chosen? Or were they the result of a form of non-conscious cognition? Or try an experiment: do a YouTube search for a supercut of sad moments from movies, and see how long you can watch it (fullscreen, music on, don’t cheat) before you begin to feel sad, or to feel your chest clench and your eyes ache a little.

Is this also unconscious cognition? And if it might be considered such, do you think there is a great difference between our sadness in response to a supercut of sad movie moments, and a neural network’s analysis of the same supercut that results in an inference that the video is “sad”? If we accept Hayles’ definition of non-conscious cognition, I am not sure that we can see human tears as particularly different from machine tears.

When people say that machines cannot weep, what they really mean is that machines don’t fully understand the data they are processing with the emotional depth of a human. The autonomous weapon system registers the presence of an enemy, compares the available data to a list of known threats, perhaps it looks for patterns to see whether the behaviour of the enemy target matches behaviour typically produced by people who have previously performed an attack, and based on this, it will either attack or not attack the target. The autonomous weapon system does not feel fear or rage or remorse, although we could certainly program it to perform some semblance of those emotions. 

But what if our emotions, based in our bodies, are no more than technical cognitive processes that might just as well be programmed in a machine? Sara Ahmed writes that “[t]hinking about what emotions do cannot be done without thinking about sweating and the sense of being in a body” (Schmitz and Ahmed 2014, 97). Machines lack bodies of flesh and blood, but they do have a material form. Is there a sweatiness of algorithmic decisions, an embodied cognition of machine learning? We tend to think of AI and machine learning as biased rather than embodied, though scholars like Wendy Chun have pointed out that the materiality of technology shapes what is possible (Chun 2008; 2017). Biases are often blamed on the datasets the machine learning is trained on. If the faces used to train a facial recognition algorithm are mostly white and male, the algorithm will be far better able to identify white men than black women (Buolamwini and Gebru 2018). Reading this through Hayles’ concepts in Unthought, one might conclude that the sweatiness, the embodiment of algorithms is in the entwinement between technical and human cognizers: “networks of non-conscious cognitions between and among the planet’s cognizers are transforming the conditions of life, as human complex adaptive systems become increasingly interdependent upon and entwined with intelligent technologies in cognitive assemblages” (Hayles 2017, 216). The biased datasets are the remnants of sweat, of human embodiment, in the machine learning. 

References

Atwood, Margaret. 1986. The Handmaid’s Tale. New York: Houghton Mifflin Harcourt.

Buolamwini, Joy, and Timnit Gebru. 2018. “Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification.” In Conference on Fairness, Accountability and Transparency, 77–91. http://proceedings.mlr.press/v81/buolamwini18a.html.

Chun, Wendy Hui Kyong. 2017. Updating to Remain the Same: Habitual New Media. Cambridge, MA: MIT Press.

Chun, Wendy Hui Kyong. 2008. “The Enduring Ephemeral, or the Future Is a Memory.” Critical Inquiry35 (1): 148–71. https://doi.org/10.1086/595632.

Drucker, Johanna. 2017. ‘Why Distant Reading Isn’t’. PMLA 132 (3): 628–35. https://doi.org/10.1632/pmla.2017.132.3.628.

Hayles, N. Katherine. 2017. Unthought: The Power of the Cognitive Nonconscious. Chicago: University Of Chicago Press.

Le Guin, Ursula. 1972. The Word for World Is Forest. New York: TOR.

Lim, Daniel. 2019. “Killer Robots and Human Dignity.” In , 6. Honolulu: ACM. http://www.aies-conference.com/wp-content/papers/main/AIES-19_paper_6.pdf.

Rogalska, Alicja, and Lukasz Surowiec. 2014. Video documentation, glass vial containing human tears. https://www.works.io/35832/tear-dealer.

Schmitz, Sigrid, and Sara Ahmed. 2014. “Affect/Emotion: Orientation Matters. A Conversation between Sigrid Schmitz and Sara Ahmed.” Freiburger Zeitschrift Für GeschlechterStudien 22 (2): 97–108. https://doi.org/10.3224/fzg.v20i2.17137.