Fractals and Hand Tracking

My Process
For the past 2-3 weeks I have educating myself on creating Generative Art in Processing. I began doing research into what already exists and I came across a great book called Generative Art, A Practical Guide Using Processing. Many of the exercises in this book went into great detail into some important terms that we will definitely be using in our project. The translate() function is a great way to move the origin point (0,0,0)  for your sketch. This is helpful because you can make fractals that can be easily spaced out without having to calculate the parent fractals location plus the location of where you want the copies. I also learned that classes are the best way to keep organize in these sometimes lengthy sketches.

The Next Step
Once I had a good understanding of how to create fractals I went looking into how to make them interactive with a user on the kinect. I referenced a tutorial sketch that I had programmed that would utilize the kinect's skeletal tracking feature and I tweaked it to draw a red ellipse onto the user's right hand. I then pulled in a branch fractal and adjusted its update location to be connected with the location of the users hand. I ran the sketch and came to the realization that it was taking up a lot of processing power to draw what the kinect was viewing and the consistently changing branch fractal so I turned off the depth image it was drawing and I told Processing to find the pixels that coincided with the user and paint those green. That made the sketch run much more smoothly. Below is a snapshot of what the sketch looked like.

Processing, the Kinect and OpenNI

The Kinect was a easy decision for our project because it is not sensitive to the light conditions in the room at the time it is captured. Hence, if we use this in a dark room it will not be an issue. The Kinect camera works by creating a depth image. It uses infrared light to create an image that captures where the objects are in space. The Kinect camera resolution is 640x480. You can bump the camera up to 1280x1024 but the data will arrive at 10 frames per second rather than 20 frames per second.

The next decision we had to make was determining which processing library would work the best for what we are trying to accomplish with the Kinect. It boiled down to OpenKinect and OpenNI.

Dan Shiffman built on the work of the OpenKinect project to create a library for working with the Kinect in processing. OpenKinect drivers provide access to the Kinect's servo motors and has a very simple software license. The contributors to the OpenKinect project released their drivers under a fully open source license. In short, this means that you can use OpenKinect code in your own commercial and open source projects without having to pay a license fee to anyone. In response to OpenKinect PrimeSense released their own software for working with the Kinect. PrimeSense included more sophisticated software that would process the raw depth image to detect users and locate the position of their joints in three dimensions. They called their software OpenNI, NI standing for "Natural Interaction". OpenNI provides two key pieces of software that is useful to our goals. First is the OpenNI framework. this includes the drivers for accessing the basic depth data from the Kinect. This piece of software has a similar licensing situation as OpenKinect. However, the other feature that OpenNI provides does not have such a simple license. This feature is the user tracking. This license is provided by an external module, called NITE. NITE is not available under an open source license. It is a commercial product that belongs to PrimeSense. PrimeSense does provide a royalty-free license that you can use to make projects that use NITE with OpenNI. 

We chose to use OpenNI because it provides us with the option to use the user tracking and there was a good amount of reading material that explained and used the OpenNI library in Processing. OpenNI also has the advantage because it is designed to work with not just the kinect but also other depth cameras. This means that code we write using OpenNI to work with the Kinect will continue to work with newer depth cameras as they are released, saving us from needing to rewrite our applications depending on what camera we want to use. 

OpenNI recognizes heads, shoulders, elbows, wrists, chests, hips, knees, ankles, and feet. This is going to be an important part in tracking the crowd in our installation. One of the techniques we are considering using OpenNI's 3D features and Point Cloud systems. Below is are screenshots of a test sketch of a point cloud system that we created that creates the illusion that it is rotating completely around what the kinect is seeing.

Rotating Point Cloud Sketch Test Image (1)

Rotating Point Cloud Sketch Test Image (2)

Research and tutorials referenced from Making Things See by Greg Borenstein

Installation size research

How much distance could we effectively use our tech in? 

Finding a balance between comfort for a group of users to experience our interactive floor with others, realistic size of fabricating the project and number of participants is a tough task. We want to let as many people as possible experience and play together, but we have to be mindful of how much room people need to stand around and be able to move their bodies. On the other hand, we also want people to feel like they are having fun together and don't want to give too much cushion between bodies. Finally, we have to realize that what we are building has to conform with the size of our lab and our budget so we can not build a studio with a stage.

Imagination can do so much, let's really see it!

As researched in our previous blog post Floor Size and Negative Space Research we have a good point of reference thanks to Dance Deck of Signature Systems Group, LLC. They are the world's largest center for renting dance floors and recommend a minimum of 4.5 square feet per person to have enough room on a dance floor. Considering this figure per person and combining it with the maximum effective distance of a X-Box Kinect which we will be using in our project we can make a good decision on the size of our floor plan.

These constraints have led us to choose a 100x100 foot square for the time being. Ideally this could populate 20 people. Unlike a traditional dance floor, our project will include display walls and a place for our performer (at this point, a DJ) to set up their materials. We have reserved a space for this which we have deemed 10 square feet, leaving 90 square feet for the participants which still equates to 20. For testing purposes we are choosing not to work in the ideal scope as we tread new waters. We are setting our maximum at 10 users until further notice and aiming for a minimum of 5 users for ideal conditions.

We want to be able to let users really go wild in our environment. Furthermore, we understand that there are limitations to the technologies we will be using during this project due to our budget restraints. All hope isn't lost friends so don't fret. Hopefully after user testing encountering the many unforeseen variables of this project we can involve more participants within this floor space, or expand our floor. Stay tuned fans! There are no instruction manuals here, just discoveries being made! #FBA!

Seeing is believing. It was crucial for us to see our floor plan in person before moving forward.

Nonverbal Behavior and Collaboration Detection

Automatically Detected Nonverbal Behavior Predicts

Creativity in Collaborating Dyads

Andrea Stevenson Won • Jeremy N. Bailenson •

Suzanne C. Stathatos • Wenqing Dai

Key Word: Rapport occurs when two or more people feel that they are in sync or on the same wavelength because they feel similar or relate well to each other. Rapport is theorized to include three behavioral components: mutual attention, mutual positivity, and coordination.

Some research indicates that pairs can be more creative than individuals working alone, but what about non verbal behavior and creativity? Rapport, or a state of mutual positivity and interest that arises through the convergence of nonverbal expressive behavior in an interaction (Drolet and Morris 2000, p.27) has been linked to success in a number of interpersonal interactions. Rapport is important to judging the success of a virtual agent.

The concept of synchronous nonverbal behavior was first introduced by Condon and Ogston (1966). Synchrony however, is very difficult to rate, and time consuming. Many people who were asked to determine the level of synchronization in videos would often revert to rating it based on similarities in people (skin color, wardrobe, etc) rather than actual actions. Eventually they would have to remove the audio from the videos that were being rated and faces were blurred. Predicting movements can also increase the amount of data that has to be processed and interpreted. Researchers have since turned to more generic ways to predict and interpret body movement.

Methods include placing of sensors on participants joints and summation of pixels for video (schmidt et al. 2012; Ramseyer and Tschacher 2011). Join markers are accurate, but can be expensive and cumbersome. Video based techniques are inexpensive, but bad lighting and bad camera angles can lessen their value. However, Microsoft Kinect is an inexpensive method that does not require joint markers with it’s infrared emitter and sensor.

Kinect was used with a teacher and learner to analyze interaction. Two kinects were used (one to record person A, one for person B) and were told to come up with as many water conservation ideas as possible. Good ideas were considered “appropriate novelty” as determined by Oppezzo and Schwartz (2014) and good ideas were marked with a 1, while bad ideas were marked with a 0 towards the final score. Overall the study predicted collaborative behavior at at a rate of 87%.

Although this study isn’t completely related to our project, it does mention troubles and solutions to using a Kinect VS other methods of motion detection as well as provides an insight into silent collaboration in a silent environment, or in our case a noisy one where communication is limited.

http://vhil.stanford.edu/pubs/2014/won-jnb-nonverbal-predicts-creativity.pdf