Programming the validation

We received some emails from blog visitors about which programs we use for the development of this set. Up to now we used Adobe Lightroom (which in the end, we didn't need at all), the Gimp, Matlab, and UFRaw. 

For the validation we had the choice of using one of several presentation programs, among which were Inquisit, E-Prime, Presentation and the Matlab Psychtoolbox. We actually decided to go with PsychoPy, a presentation library for Python, which makes the validation program platform independent and open source. Convenient, when you program on OS X or Linux, but need to run on Windows XP.

We needed some kind of intelligent way of distributing all images across participants, so that everybody rates the same amount of models, with the same amount of different emotions and eye gazes. Because of this, we first programmed a sequence generator in Python that generates the sequences of stimuli any participant will see before the validation even starts. Our script utilizing PsychoPy subsequently reads these sequences and shows the appropriate images according to someone's participant number. We had to program the functions to show likert scales or open ended questions ourselves, but hey, Python is fun. 

Monday, we start with data collection. After this, we analyze the ratings and will create the infrastructure to distribute the set. We'll keep you posted.

Ron

Small comment: lightroom and too intelligent cameras

Dear all,

just a small sidestep on current work. We currently are busy with defining the cropping parameters for all images in lightroom in order to finally implement the optimal alignment parameters mentioned in the previous post.

Thereby I stumbled upon an interesting issue: when you define in the lightroom files that you want to crop e.g. the top of the image, what actually is cropped can differ from image to image. Searching a bit deeper, I found a "Camera Orientation " setting in the settings files and a page about intelligent camera orientation sensors on the web.

So, cameras nowadays know, how they are oriented, when a photo was taken. And obviously, lightroom knows and reads this intelligent orientation parameter to present all pictures upright. Nevertheless, it still defines 'top' for cropping relative to the camera housings top. Now, is that clever, or what?

About aligning landmarks of the RaFD faces

Dear all,

I shortly will describe in this post, how we solved the problem of aligning landmarks of all face images for the RaFD. Making a database with really good alignment of eyes, nose etc. seemed germain to us, as for our own research we had to align the images of other databases regularly manually.

The problem
Even though we paid close attention to the positioning of our models during the photoshoot, the images throughout the session show considerable variation in positioning:


The solution
As you can imagine, doing a manual alignment for 17.000+ pictures is not really an option (and not really objective either). So we opted for an automated procedure for this and came up with a rather simple solution. We currently use the nonlinear general purpose fitting function 'fminsearch' from Matlab to optimize the correlation between a template and a target image. The optimization procedure varies two translational parameters (x-, and y-direction) plus one rotational parameter. For each iterative step, the correlation of template and target is calculated for a specific area of the images. fminsearch then varies those parameters until some matching criterion is met.

To see it working, look at this:

Creating standardized expressions for the RaFD

Target Face Action Units

There are lots of ways that one can express the same emotion through the face, and there are, of course, huge individual differences based on physiology, prior personal experiences, and culture, just to name a few factors. However, one of the main goals for the RaFD is to produce a standardized set of facial expressions in which people are more or less using the same facial muscles to convey the emotion. This has obvious benefits for psychological research - in facial mimicry studies, for example, this standardization means that you can be sure that participants are mimicking something that is actually there in every picture being shown to them. To my knowledge, there is currently no other existing database of static images that has attempted to have standard facial expressions available for such a large number of models.

When developing the faces used for the RaFD, Job van der Schalk and I based our target expressions upon prototypes defined by Paul Ekman and Walter Friesen, and attempted to elicit these expressions through a variation of the Directed Facial Action Task (DFAT)1. Our knowledge of the Facial Action Coding System (FACS) was invaluable to this process.

Below are the target codes (in terms of FACS Action Units) that we attempted to elicit from the models. AUs in boldface are "core AUs" that we considered to be absolutely essential to photographing before moving on to the next emotion, and were the main requisites for selecting pictures used in pilot tests.

Anger: 4CDE + 5CDE + 7 + 17 + 23/24
Contempt: Unilateral 14
Disgust: 9 + 10 + 25
Fear: 1 + 2 + 4 + 5DE + 20 + 25
Happiness: 6 + 12CDE + 25
Sadness: 1 + 4 + 15ABC + 17
Surprise: 1 + 2 + 5AB + 26

References

1. Ekman, P. ( 2007). The directed facial action task. In J. A. Coan and J. J. B. Allen (Eds.), Handbook of Emotion Elicitation and Assessment (pp. 47-53). Oxford University Press.

Coaching Models in the Photo Session

We typically had about 45 minutes to collect all of the desired photos from each model. This amounted to about 6 minutes per emotion, on average. Some poses were accomplished relatively quickly (such as joy), while more complex expressions (i.e., fear and sadness) sometimes took up as much as 1/2 of the total session time. There were at least 3 individuals working with the models during the shoots: 1) a FACS trainer who coached models on the AUs, and took photos using a remote control when the expression was satisfactory; 2) A posture checker sitting in front of 5 computer monitors, making sure that the model's head and body were correctly in frame from every angle; and 3) a posture coach stading behind the FACS trainer, who relayed information from the posture checker to the model. This allowed the FACS trainer to concentrate solely upon the face without also worrying about giving posture instructions.

Posture Checker (Oliver Langner) and FACS Trainer (Skyler Hawk)
Photograph by Bert Meelberg

FACS Trainer (Skyler Hawk) Posture Coach (Gijs Bijlstra)
Photograph by Bert Meelberg

Job van der Schalk and I have shared interests and backgrounds in dramatic arts, including acting and directing, that were essential to our jobs as FACS coaches. Not only did we have to elicit the correct AUs from models, but it was important to stay energetic, give constructive feedback, and keep them in good spirits. It's important to note that this whole process could be quite exhausting for the models, both physically and mentally. We tried to keep things going smoothly by giving short breaks, if needed. When models had trouble making certian AU combinations during the shoot, we provided further on-site instruction with the use of a hand-held mirror.

Sometimes, the exhaustion that can come with exercising the facial muscles in this way required us to move on to another expression that used completely different sets of AUs, coming back to the former expression as time allowed. This was especially true if models didn't already use these AUs in their own expressions of certain emotions. We found that 1+4 and 1+2+4 combinations required the most sustained effort, although lower-face AUs such as 15 and 20 also tended to "fade out" without continued feedback to keep things at the desired strength.

Training Models

Model Training and Rehearsal

Models typically received a photo training manual at least 24 hours in advance. This manual contained both written instructions and sample pictures drawn from two different existing databases (the JACFEE1 and Karolinska2 sets). We requested that models practice for at least an hour prior to their appointment, whenever possible.

Before beginning with the actual photo session, we made sure that models were able to make the expressions as requested by leading them through warm-up and refinement training exercises. This rehearsal session typically lasted 30 minutes. This allowed us to assess the initial capabilities of the models and to get an idea of which expressions would need more time and effort during the photo shoot. It also allowed us to try several different "tricks" that we developed, beyond the DFAT instructions, to elicit the tagret AUs. Some of these tricks worked better than others, but which ones were best really depended on the individual. Some people responded better imagery-based tasks, for example, while others were more assisted by physiology-based instruction. If you have further comments/questions about our coaching/training procedures, be sure to add a comment.

References

1. Biehl, M., Matsumoto, D., Ekman, P., Hearn, V., Heider, K., Kudoh, T., & Ton, V. (1997). Matsumoto and Ekman's Japanese and Caucasian Facial Expressions of Emotion (JACFEE): Reliability data and cross-national differences. Journal of Nonverbal Behavior, 21, 3-21.

2. Goeleven, E., De Raedt, R., Leyman, L., & Verschuere, B. (2008). The Karolinska Directed Emotional Faces: A validation study. Cognition & Emotion, 22, 1094-1118.