– by Amy Zhao and Brendan Zietsch

Facial attractiveness studies have typically relied on asking people to rate facial photos of real-life participants or images of computer-generated faces. However, these ratings can be subjective and affected by rater biases. More recent studies (such as our own) have attempted to avoid subjective biases through the use of facial landmarks to derive objective measures of facial traits. However, these landmark-based measures ignore features thought to be relevant to face perception such as skin colour and contrast, hair, and eye colour. Here, we introduce deep neural networks as a method that combines the strengths of both approaches while addressing the limitations of facial landmarks.

Facial recognition neural network models are designed to extract abstract facial features from images. Each image input yields one set of multidimensional coordinates in feature space — a space representing the compressed version of the original image, with the number of dimensions representing the number of abstract facial features. The distance between two points (i.e. faces) in feature space reflects facial similarity, with similar faces represented by points that are closer together. While these coordinates lack direct interpretability, they effectively quantify abstract facial qualities that can be used to calculate facial traits relevant to facial attractiveness research.

We applied an existing facial recognition neural network model (VGG16) to facial images from our speed-dating study (n = 682). We used the extracted feature space coordinates to calculate traits such as facial averageness, similarity, and masculinity to predict in-person ratings of facial attractiveness and kindness. We then compared this neural network method to traditional manual (and automatic) landmark methods.

An issue that has been alluded to in past studies is that landmark measures of masculinity could be influenced by facial pitch (upward or downward tilt of a face). In our images, men tended to tilt their heads upward compared to women (there was a significant difference in facial pitch angle between genders). We found facial pitch was highly correlated with landmark measures of masculinity (-.17 ≤ r ≤ -.73). In contrast, there was little to no correlation between facial pitch and neural network measures of masculinity (.00 ≤ r ≤ -.23). Likely, gender differences in the way that men and women pose for photos might bias typical landmark masculinity measures. Here, we demonstrate that neural networks can extract facial information without being affected by limitations associated with landmarks.

Overall, facial measures derived from neural networks predicted in-person ratings, largely replicating what we found in our previous study using manual landmarks. Some differences were that neural network measures of masculinity robustly predicted facial attractiveness in men, whereas there was only context-dependent evidence for this in our previous study. We also found novel evidence for assortative preferences for facial masculinity. For example, participants with sex-atypical faces (a masculine woman or feminine man) revealed stronger preferences for a partner who was sex-typical (i.e. they rated sex-typical partners more attractive) than those with sex-atypical faces. We believe that we saw such effects in the context of neural network measures of masculinity due to increased visual information that was extracted from images of participants as well as decreased noise from participant facial pitch.

Neural network-derived measures had small to moderate correlations with landmark-based measures (.11 ≤ r ≤ .33), while manual and automatic landmarks were moderate to strongly correlated as expected (.29 ≤ r ≤ .86). Neural network masculinity measures were more accurate when it came to classifying the sex of the participant (95.6 % ≤ accuracy) compared to landmark measures (75.3% ≤ accuracy ≤ 88.8 %). Both low correlations between neural network and landmark measures as well as relatively higher sex-classification accuracy from neural network measures suggest that there is relevant information from facial photos that is uncaptured by landmarks. However, we did not find that neural networks were better (explained more variance) at predicting in-person ratings compared to other landmark measures.

While we found that neural network-derived measures do indeed predict in-person ratings, the underlying method is not well understood. Unlike landmarks, where we understand that the “average” male is one that most resembles the facial structure (as described by landmark coordinates) of the average male face, we are unclear as to what the “average” neural network face resembles – that is, what aspects of the face are contributing more or less to its position in feature space. While there are some ways in which we can use landmarks to describe (and visualise) shape variation, we are unaware of any straightforward way to visualise variation in feature space coordinates. (We note that we did create composite images of the top 20 participants for each trait.) While we controlled for ethnicity variables, this does not mitigate any systematic biases that may arise from imbalances in training face recognition models used by automatic landmarks and neural networks.

Given the lack of transparency behind neural network models, we suggest that researchers use caution when employing these methods. However, we also believe that neural networks are a fast, reproducible, and powerful way to extract visual information without the limitations associated with landmarks. A link to instructions and code for obtaining feature space values using neural networks is available in the full text of this paper.

Read the original paper: Zhao, A.A.Z., & Zietsch, B. (2024). Deep neural networks generate facial metrics that overcome limitations of previous methods and predict in-person attraction. Evolution & Human Behavior, 45(6), 106632.

– by Oliver Sng & Krystina Boyd-Frenkel

If you think about the last stranger you met, you will likely remember their race. But why do we care and think about others’ race? Laypersons sometimes suggest that humans have evolved to be racist. Evolutionary scholars know that the truth is far more complex. Instead, the somewhat surprising answer (at least to laypersons) is that, without modern transportation, our ancestors rarely encountered individuals who looked phenotypically different enough to qualify as a different ‘race’. As a result, natural selection couldn’t have shaped a psychology for interacting specifically with different races.

Why, then, do people care about others’ race today? One prominent answer has been that race is a cue to coalition—the groups that we work with, compete against, and generally solve life’s problems with. Our families, work colleagues, and political groups are all examples of coalitions. Considerable research has accumulated supporting the race-as-coalition perspective, including replications and re-analyses. Our recent work offers a second, complementary answer to this question: that race is a cue to ecology.

The “race-as-ecology” perspective proposes that people pay attention to and think about others’ race because people assume different races live in different (social) ecologies. In the U.S., people assume that Black individuals, relative to White individuals, live in harsher ecologies. Living in harsh ecologies – where early death from unavoidable causes like disease or violence is common – has been linked to traits including an earlier age of first reproduction, a more present-focused time perspective, and less investment in skill accumulation (e.g., education) (see two recent reviews here and here). If living in harsher ecologies influences people’s behaviors, then knowing about another person’s ecology may help us understand and predict their behavior.

We are aware of the ongoing debates in the life history literature, with important issues such as exactly why and how people respond to harsh ecologies, or whether life history “strategies” exist in our species. However, these do not necessarily affect our current work, as we focus on people’s perceptions of others, based on what people think others’ ecologies are. From our perspective, as long as (1) individuals living in harsher ecologies adopt different behaviors, and (2) certain racial groups are presumed to be living in ecologies of varying harshness, then it follows that (3) people will categorize others by their racial group because they assume different racial groups to be living in ecologies of different harshness. If so, one critical implication is that when individuals of different racial groups are presented as equally living in harsh ecologies (or not), people should care about their race less. In the presence of direct information about another’s ecology, their race isn’t useful information anymore. This is the essence of the race-as-ecology perspective.

In a set of three studies, we test these implications. Using a widely used method in the literature, sometimes referred to as the “who-said-what” method, American participants viewed photos of Black and White individuals paired with sentences presumably spoken by each person. Each Black or White person was randomly presented multiple times. After that, participants were given a surprise memory test, in which they were shown the sentences again, but now tried to remember who said each sentence. In our research, what matters are the mistakes participants make. If a participant misremembered what a Black person said as being said by a different Black person, they essentially confused the two Black individuals (a within-race confusion). However, if the participant misremembered what a Black person said as being said by a White person, they have instead confused two individuals of different races (a between-race confusion). More within-race (vs. between-race) confusions indicate that a person categorizes others by race, mentally grouping Black individuals together and White individuals together.

In our studies, half of our participants saw Black and White individuals presented with just their faces. The critical manipulation is that the other half of our participants saw the same Black and White individuals, but now presented in the ecologies/neighborhoods that they supposedly live in. Importantly, both Black and White individuals were presented evenly in both relatively harsh (or the opposite, referred to as “hopeful”) ecologies (see example photos below).

Sample race-with-ecology photos used in studies

What do we observe? First, participants categorized these individuals by their ecologies. In other words, they were more likely to confuse individuals living in harsh (or hopeful) ecologies with other individuals also living in harsh (or hopeful) ecologies. Second, participants categorized these individuals by their race, but they did so less when the Black and White individuals were shown in both harsh and hopeful ecologies. Hence, ecology information leads to the reduction of racial categorization.

Could just telling people to not pay attention to others’ race have the same effect? Past research has tried this and failed. In fact, getting people to stop thinking about race isn’t easy. The current “race-as-ecology” perspective provides insights into one way in which this can be achieved, complementing work from the “race-as-coalition” perspective.

There are puzzles that remain, and new puzzles that emerge. For example, even when race is paired with ecology, we do not see racial categorization completely disappearing (as is sometimes observed in work from the race-as-coalition perspective). So, people are still categorizing others by their race in the presence of ecology information. This suggests that other processes (like race-as-coalition) are still at play.

So, can race be “replaced”? Our answer is a partial yes. When people see others of different races, but living in different ecologies, they group others by their ecologies. In the minds of perceivers then, race is replaced by ecology. In other recent work, we also find that people hold ecology stereotypes—general beliefs about what individuals who live in harsh ecologies are like—and that these stereotypes exist across multiple societies. To the extent that people around the world think about others in terms of their ecology, there may be a range of other social categories, beyond race, that also have the potential to be “replaced” by ecology.

Read the original article: Sng, O., Boyd-Frenkel, K. A., & Williams, K. E. G. (2024). Can race be replaced? Ecology and race categorization. Evolution and Human Behavior, 45(6), 106630.