Fractals and Hand Tracking

My Process
For the past 2-3 weeks I have educating myself on creating Generative Art in Processing. I began doing research into what already exists and I came across a great book called Generative Art, A Practical Guide Using Processing. Many of the exercises in this book went into great detail into some important terms that we will definitely be using in our project. The translate() function is a great way to move the origin point (0,0,0)  for your sketch. This is helpful because you can make fractals that can be easily spaced out without having to calculate the parent fractals location plus the location of where you want the copies. I also learned that classes are the best way to keep organize in these sometimes lengthy sketches.

The Next Step
Once I had a good understanding of how to create fractals I went looking into how to make them interactive with a user on the kinect. I referenced a tutorial sketch that I had programmed that would utilize the kinect's skeletal tracking feature and I tweaked it to draw a red ellipse onto the user's right hand. I then pulled in a branch fractal and adjusted its update location to be connected with the location of the users hand. I ran the sketch and came to the realization that it was taking up a lot of processing power to draw what the kinect was viewing and the consistently changing branch fractal so I turned off the depth image it was drawing and I told Processing to find the pixels that coincided with the user and paint those green. That made the sketch run much more smoothly. Below is a snapshot of what the sketch looked like.

Nonverbal Behavior and Collaboration Detection

Automatically Detected Nonverbal Behavior Predicts

Creativity in Collaborating Dyads

Andrea Stevenson Won • Jeremy N. Bailenson •

Suzanne C. Stathatos • Wenqing Dai

Key Word: Rapport occurs when two or more people feel that they are in sync or on the same wavelength because they feel similar or relate well to each other. Rapport is theorized to include three behavioral components: mutual attention, mutual positivity, and coordination.

Some research indicates that pairs can be more creative than individuals working alone, but what about non verbal behavior and creativity? Rapport, or a state of mutual positivity and interest that arises through the convergence of nonverbal expressive behavior in an interaction (Drolet and Morris 2000, p.27) has been linked to success in a number of interpersonal interactions. Rapport is important to judging the success of a virtual agent.

The concept of synchronous nonverbal behavior was first introduced by Condon and Ogston (1966). Synchrony however, is very difficult to rate, and time consuming. Many people who were asked to determine the level of synchronization in videos would often revert to rating it based on similarities in people (skin color, wardrobe, etc) rather than actual actions. Eventually they would have to remove the audio from the videos that were being rated and faces were blurred. Predicting movements can also increase the amount of data that has to be processed and interpreted. Researchers have since turned to more generic ways to predict and interpret body movement.

Methods include placing of sensors on participants joints and summation of pixels for video (schmidt et al. 2012; Ramseyer and Tschacher 2011). Join markers are accurate, but can be expensive and cumbersome. Video based techniques are inexpensive, but bad lighting and bad camera angles can lessen their value. However, Microsoft Kinect is an inexpensive method that does not require joint markers with it’s infrared emitter and sensor.

Kinect was used with a teacher and learner to analyze interaction. Two kinects were used (one to record person A, one for person B) and were told to come up with as many water conservation ideas as possible. Good ideas were considered “appropriate novelty” as determined by Oppezzo and Schwartz (2014) and good ideas were marked with a 1, while bad ideas were marked with a 0 towards the final score. Overall the study predicted collaborative behavior at at a rate of 87%.

Although this study isn’t completely related to our project, it does mention troubles and solutions to using a Kinect VS other methods of motion detection as well as provides an insight into silent collaboration in a silent environment, or in our case a noisy one where communication is limited.

http://vhil.stanford.edu/pubs/2014/won-jnb-nonverbal-predicts-creativity.pdf