Perceived Realism: Measuring Neural Response to Virtual characters with EEG

As virtual human characters continue to grow omnipresent in movies and games, the ability to create realistic virtual characters becomes a prescient commodity for animation artists. Because animation artists must create virtual characters that are human-like to be believable but not highly realistic to turn people off.

For example, studies have found that people do not like virtual characters that are highly realistic (i.e., too good to be true). This phenomenon is known as the uncanny valley effect. The uncanny valley effect was first proposed by Masahiro Mori in 1970. He argued that the believability of virtual human characters begins to wane as characters become highly realistic. Sometimes these characters may evoke revulsive feelings, but more so when characters are mobile than when stationary.

Researchers have delved into the uncanny valley effect, and whereas the phenomenon is established and widely known among animation artists, the methods used to measure this effect are somewhat subjective. Some researchers observe the facial expressions (either pleasant, neutral, or revulsive) of people as they watch virtual characters while others have asked participants to rank the humanness of virtual characters using the Likert scale. Findings from these studies, however, tend to be highly subjective and sometimes unreliable.

A team of researchers (Mustafa et al., 2017), interested in this phenomenon and knowing the neural response underlying facial expressions, designed an experiment to measure people’s neural response to virtual human characters using the electroencephalogram (EEG). The EEG, compared to other devices such as fMRI is an ideal tool for measuring the neural response because it has reliably been used in cognitive neuroscience research, “can detect changes within a millisecond time-frame, and allows monitoring of the information processing during stimulus presentation”.

Two studies were conducted: an online perceptual study and an EEG study. The purpose of the online study was to allow participants to rate the humanness of characters using a Likert scale. The EEG study, on the other hand, measured participants’ neural responses as they watched real human and computer-generated human characters. This experiment aimed to measure the neural response to real humans and computer-generated human characters to offer an objective experimental design to measure the uncanny valley effect.

Whereas 80 participants completed the online perceptual study, forty participants (including 22 females and 18 males with an average age of 24) completed the EEG study. None of these participants had a neurological disorder that could have confounded the results.

Materials used for the study were obtained from the Institute of Creative Technologies Virtual Human Toolkit. As well as computer-generated humans Digital Emily and Digital Ira. For the highly realistic characters, the authors used characters from interactive drama video games, ‘Kara’ from Detroit: Become Human, ‘Ernst’ from Squadron 42, and ‘HeadTech’ from Janimation.

As hypothesized, people’s ranking of humanness (and sometimes the likeness of the virtual characters) began to dip as the characters become highly realistic.

Figure 1: This initial proposal by Mori shows how “the more human-looking a character becomes the more agreeably it is perceived until the character becomes so nearly human that it causes a response of revulsion, as seen by the valley where the corpse and zombie end up before the response becomes positive again in response to real humans”.

Similarly, the neural response from participants in this study show a similar trend in people’s perception of the humanness of a character

Figure 2: “Probabilities of an uncanny neural response plotted against the user responses from the perceptual study with a fourth-order polynomial fitted against the data points. The results show a correlation between the neural responses classified by the SVM and how realistic each agent is. The more human-like a computer-generated character, the more the neural response tends towards the ‘uncanny’. The results resemble Mori’s uncanny valley hypothesis as seen in Fig. 1”

Data obtained from the brain activities showed different responses as participants watched the different human characters. That’s, highly realistic computer-generated characters elicited the higher neural response (depicting participants revulsion) whereas real human characters produced “normal” neural response. The online perceptual study also had similar results with Emily (the highly realistic character) perceived as an uncanny character.

In sum, this study confirms the uncanny valley effect first proposed in 1970, but most importantly, it also offers an objective methodology to study this phenomenon using the EEG.