Nonverbal Behavior and Collaboration Detection

Automatically Detected Nonverbal Behavior Predicts

Creativity in Collaborating Dyads

Andrea Stevenson Won • Jeremy N. Bailenson •

Suzanne C. Stathatos • Wenqing Dai

Key Word: Rapport occurs when two or more people feel that they are in sync or on the same wavelength because they feel similar or relate well to each other. Rapport is theorized to include three behavioral components: mutual attention, mutual positivity, and coordination.

Some research indicates that pairs can be more creative than individuals working alone, but what about non verbal behavior and creativity? Rapport, or a state of mutual positivity and interest that arises through the convergence of nonverbal expressive behavior in an interaction (Drolet and Morris 2000, p.27) has been linked to success in a number of interpersonal interactions. Rapport is important to judging the success of a virtual agent.

The concept of synchronous nonverbal behavior was first introduced by Condon and Ogston (1966). Synchrony however, is very difficult to rate, and time consuming. Many people who were asked to determine the level of synchronization in videos would often revert to rating it based on similarities in people (skin color, wardrobe, etc) rather than actual actions. Eventually they would have to remove the audio from the videos that were being rated and faces were blurred. Predicting movements can also increase the amount of data that has to be processed and interpreted. Researchers have since turned to more generic ways to predict and interpret body movement.

Methods include placing of sensors on participants joints and summation of pixels for video (schmidt et al. 2012; Ramseyer and Tschacher 2011). Join markers are accurate, but can be expensive and cumbersome. Video based techniques are inexpensive, but bad lighting and bad camera angles can lessen their value. However, Microsoft Kinect is an inexpensive method that does not require joint markers with it’s infrared emitter and sensor.

Kinect was used with a teacher and learner to analyze interaction. Two kinects were used (one to record person A, one for person B) and were told to come up with as many water conservation ideas as possible. Good ideas were considered “appropriate novelty” as determined by Oppezzo and Schwartz (2014) and good ideas were marked with a 1, while bad ideas were marked with a 0 towards the final score. Overall the study predicted collaborative behavior at at a rate of 87%.

Although this study isn’t completely related to our project, it does mention troubles and solutions to using a Kinect VS other methods of motion detection as well as provides an insight into silent collaboration in a silent environment, or in our case a noisy one where communication is limited.

http://vhil.stanford.edu/pubs/2014/won-jnb-nonverbal-predicts-creativity.pdf