Faces and Interfaces

* A lot of this material is taken from Kyle McDonald’s course, check it out for a deeper and more technical overview.

The new generation of iPhones ships with advanced face recognition and tracking capabilities. You are be able to unlock your phone by looking at it, and to “Pay with your face”

Humans are extremely good at recognizing and reading faces. Face perception has played a central role for social interaction for millions of years, informing us about the identity, age, gender, mood, attractiveness, race and friendliness of a person.

Face on Mars

One of the most salient objects in our day-to-day life is the human face. Faces are so important that the impairment of our face-processing ability is seen as a disorder, called prosopagnosia while unconsciously seeing faces where there are none is an almost universal kind of pareidolia.

Faces in things twitter

[From Wikipedia] “Chernoff Faces” are an information visualization technique, invented by Herman Chernoff in the early 1970’s, which represents multivariate data in the shape of a human face. The individual parts, such as eyes, ears, mouth and nose represent values of the variables by their shape, size, placement and orientation. The idea behind using faces is that humans easily recognize faces and notice small changes without difficulty. Chernoff used 18 variables (such as eyebrow height, jaw size, etc.) to describe a face.


Chernoff Faces

Structurally, Chernoff faces use 18 dimensions.

Weird faces by Moka, an algorithm to imitate his own hand-drawings:


Our ability to read faces from minimal information is leveraged by character designers all the time.

Face detection, recognition, and tracking

When talking about computers vision and faces, there are three major modifiers on the word “face”: detection, recognition, and tracking. Each of these has a specific meaning, though they are sometimes used loosely:

Face Detection

  • Detection is a binary classification task. That means it’s only about deciding whether something is a face or not. Detection means knowing the difference between a face and a non-face.

Eigen Faces

  • Recognition is about association, or naming. Recognition means knowing the difference between my face and your face.

Face tracking with ASM

  • Tracking is about detection, and possibly recognition, over time. Tracking means knowing that the face in this frame of video is the same as the face in the last frame of video.

When you start talking about properties of the face itself, you will also hear:

Pose estimation

  • Pose describes the position and orientation of the face.

  • Expression describes the gesture the face is making.


Not just for fun

Early pre-standardization mugshots by the private detective agency Pinkerton
Pre-standardization mugshot

Face identification has always been an interest of law enforcement, which drove the research into computer assisted face tracking.

Despite the fact that other methods of identification (such as fingerprints, or iris scans) can be more accurate, face recognition has always remains a major focus of research because of its non-invasive nature and because it is people’s primary method of person identification. (Tanzeem Choudhury)

In 1988/89 researchers developed techniques for automatically recognizing faces if the face was already aligned and normalized. They used techniques like principal components analysis, and neural networks for classification. In 1991 this technique was extended to face detection.

FERET Excerpt

Face recognition became especially controversial in 2001:

The technology first captured the public’s attention from the media reaction to a trial implementation at the January 2001 Super Bowl, which captured surveillance images and compared them to a database of digital mugshots. (US Government Subcommittee on Biometrics)

Face Finder from Visage

Visionics Face DB

Wired has a good article with some thoughts on whether it was constitutional, and relating some of the fears at the time.

Face recognition is becoming ubiquitous, being able to recognize people from pictures and video feeds, consensually or not, can create a lot of value from targeted marketing (tracking customers’ behavior) and mass surveillance (private data can usually be accessed by governments).

Face recognition technologies are increasingly being connected with machine learning to seek for patterns or create more complex mappings.

If the datasets are flawed these systems can reproduce and reinforce stereotypes. Garbage in -> garbage out

Questionable goal, questionable methods for this “gaydar” research


Artists messing with face recognition technologies

CV Dazzle

Facial weaponization suite by Zach Blas

Face expression transfer