Phase 3: Emotion and the Uncanny Valley

Exploring incongruent emotional expressions

In my last phase, I wanted to take my investigation of the uncanny valley effect in a different direction. I had been reading a lot of research into the effect of categorical perception and congruity on how people perceive near-human agents, and I had started to be interested in how we convey and understand emotion through facial expressions. One particularly fascinating study was Tinwell, Nabi and Charlton’s (2013) work on linking perceptions of psychopathy in humans with certain patterns of emotional expressiveness in different parts of the face. Their research presented participants with a video of a near-human agent reacting to a startling sound. Agents were perceived as eerier when they only showed an emotional response in the lower part of their face, while the upper part remained static. I had also observed that many of the eerier near-human agents in my own research were those with exaggerated or distorted eyes, and had noticed that many eerier computer game characters were those where the expressions in the eyes were blank or not convincing. I wondered if I would be able to demonstrate this experimentally if incongruent expressions were presented in the eye region and the rest of the face: would very happy, angry, disgusted, frightened or sad faces with ‘dead’ eyes really be rated as the most eerie?

To do this, I created a suite of images from photographs of volunteers who had been trained to pose emotional expressions. For each model, I swapped the eye region into the base face for different combinations of emotions. A table of all the different combinations is shown below.

I had four broad areas of research to explore:

• Would particular combinations of mismatch turn out to be eerie after all?

• If I measured the level of fear, anger, sadness or disgust people felt when looking at each of the combinations, which part of the face would drive the response?

• Would people be able to identify the emotions in the faces?

• Would people be able to accurately classify the mismatched faces as displaying positive, negative or neutral expressions?

Eeriest mismatches

I had expected that the eeriest faces would be 4 and 6A-6S in the middle row, showing strong emotional faces with blank eyes. This would go with Tinwell et al’s hypothesis and also my own observations of eerie ‘dead-eyed’ near-human agents. However, this was not the case. The eeriest faces were actually 7A and 7F - the very happy faces with angry or frightened eyes.

Emotional responses

I found that the emotions reported by participants were strongest for the faces where the expressions were not blended, and were clearly and unambiguously presented. A full visualisation of all the results is shown below:

Conclusions

The key finding that I took away from this research phase was that the faces that evokes the strongest responses of eeriness were those where the mismatched expressions indicated that the viewer really shouldn’t trust the entity that they were engaging with. The faces where a happy lower face was mismatched with angry and fearful eyes were the ones rated as eeriest, and in the context of the uncanny valley, this would suggest situations where the entity you’re encountering is attempting to conceal an emotion that could indicate a threat to you as the viewer. Linking this back to earlier research into either psychopathy or masked expressions suggests that there is an essential disconnect with the viewer’s ability to emphasise with and trust the person they are talking to. This has intriguing possibilities for future directions in creating trustworthy, believable artificial agents - if we’re going to experience these more in our everyday lives, we need to be able to feel that we’re safe with them. This is especially key in contexts where we might encounter artificial agents in a service, supportive or caring role.

These findings are currently being prepared for publication.