This tool can record all the data coming into an application from a Kinect unit. You can then view, review and store the data. Kinect Studio lets you inject the captured data streams back into a Kinect-enabled application, allowing you to test your code without getting out of your chair (although it might be healthier to make developers get up and move). If you’re working on a project in a distributed team, you can easily share data files with team members, enabling consistent testing across the team.

To begin, open the solution for the Kinect application and start it in debug.

While the application is running, start Kinect Studio. As shown in Figure 1, Kinect Studio has four windows: a main window and Color Viewer, Depth Viewer and 3D Viewer windows. The main window shows a timeline as well as the controls used in Kinect Studio. In the Color Viewer window, you can see the couch in my living room. The Depth Viewer window uses color to represent the distance that an object or a person is from the Kinect unit. Red indicates a distance closer to the Kinect unit, and blue represents a distance farther from it. Using color to represent distance provides a visual indication of the distance. In the 3D Viewer window in Figure 1, notice that the view is slightly rotated with respect to the images in the Color Viewer and the Depth Viewer.

Capturing Kinect Data

When Kinect Studio is first started, the Connect to a Kinect App & Sensor dialog box shown in Figure 2opens.

In this dialog box, you specify which Kinect-enabled application you want to connect to. Connecting to an application enables Kinect Studio to capture the data coming into that application from the Kinect unit. Based on the windows available, Kinect Studio captures the data feeds from the color stream and the depth stream. With the application running and Kinect Studio connected, you can now capture the data. In the Kinect Studio main window, click the Record button (or press Ctrl+R) to begin collecting data. Next, have the test subject move through the scenario you want to test. When the scenario is complete, click the Stop button (or press Shift+F5). Kinect Studio then stores the data in memory. Once Kinect Studio is done processing the data, the timeline on the main window populates, as do the Color Viewer, Depth Viewer and 3D Viewer windows, as shown in Figure 3.

By moving the cursor along the timeline, you can see the content relative to the timeline selection. Typical of most video editing software, sections of the timeline can be selected and saved as a separate file so you can use just the parts you want. The file saved is an .xed file. All the collected data is now contained in the file, enabling you to replay it whenever you want or distribute it to the rest of your team.

Notice that there isn’t a Skeleton Viewer. Kinect Studio doesn’t capture skeleton data (collection of joints) because that data is evaluated at run time based on the depth and color views. Capturing a skeleton view would defeat the purpose of recording the data for testing. In other words, the intrinsic data comes from the depth and color sensors. The skeleton data is the product of the Kinect for Windows software analyzing this data. Therefore, the runtime is going to re-evaluate the skeleton data from the depth and color data being injected back into the Kinect-enabled application under development just as though a user were in front of the Kinect unit providing live interaction.

Now let’s see just how useful this tool is. While the Kinect-enabled application is running, open Kinect Studio, connect to the application and open the .xed file you previously saved. When you click the Play button (or press F5), Kinect Studio injects the data saved in the .xed file into the Kinect-enabled application, simulating the user (or users). The application in development reacts to the .xed feed as though the user were actually doing the actions in front of the Kinect in real time. In addition to relieving you from having to hop in front of the Kinect unit every time you want to test a change you make to your application to test it, this capability lets you test the application for different-sized users.

Capabilities of Kinect Studio

Let’s look at some of the other features Kinect Studio offers. While using the timeline to set the image where desired, the Color Viewer window shows a color image of the resolution that you enabled for the color stream from the Kinect. You can right-click on the image and select “Save image as” to save the still image as a bitmap.

The Depth Viewer also has a useful utility. To use it, pause the video at a specific point and then move the mouse pointer across the image. At the bottom of the Depth Viewer window, notice the data points that are displayed. First, the frame number indicates the frame number being displayed. Next, the x,y coordinates relative to the image being displayed are shown. Finally, and most interesting, the distance in millimeters from the Kinect to the object or person that the mouse pointer is “touching” is given. Let’s look at an example to understand this information a little better. I asked my daughters (Kenzy and Sherrie) to help me with this demonstration. I used Kinect Studio to capture them doing a routine that involved moving their arms and legs. Figure 4 shows how they were positioned: Kenzy is slightly behind but to the side of Sherrie.

In the image on the left, the mouse is on the point x=178, y=304. The depth is 2187 millimeters (mm). So this point (which corresponds to the “girl in red” [Sherrie]) is 2187 mm from the Kinect unit. The image on the right shows the mouse on the point x=326, y=304, with a depth of 2639 mm. So this point (which corresponds to the “girl in orange” [Kenzy]) is 2639 mm from the Kinect unit. Another way to interpret this data is to say that Kenzy is 452 mm behind Sherrie. The Kinect depth sensor provides this data and is one of the features that makes Kinect extremely powerful.

We have had the ability to capture data from webcams and similar hardware for many years. Kinect gives us the ability to capture the distance of objects in a relatively inexpensive package. By combining these data streams, we can create applications that “understand” not only the imagery being presented but also the three-dimensional aspects of the scene. Think about when relational databases first became commonplace. The ability to relate two seemingly disparate pieces of data via a defined relationship revolutionized the way applications work and the capabilities they can offer. Similarly, using Kinect, we can now draw far more value from the scene in front of the Kinect and can relate objects through analysis of imagery as well as actual distances.

Let’s take another look at Figure 4 from the perspective of developing an application that wants to know which user is closest to the screen in order to select the main player, for example. If all I had to work with was a color image of two children, it would be difficult to ascertain which child was standing closer to the screen. In fact, I would probably have to use some other means of determining which child was the main player, such as having the player highlight and select the shape of her body outline. Using the depth data, however, I no longer need to have these artificial mechanisms for determining what is obvious based on the physical layout. With Kinect, we’re a step closer to ubiquitous computing because the application is able to infer information by evaluating the user’s situation without requiring direct, artificial interaction (such as clicking the mouse or typing on a keyboard). The user simply behaves naturally. That’s the real power of programming Kinect-enabled applications. Developers can build applications for average businesses and households that allow users to be themselves. Or as Microsoft puts it, “You are the controller.” Without the depth data, this would not be feasible.

For me, the 3D Viewer window is the coolest feature of Kinect Studio. Again, using the timeline to set the point in time that you want to view, the image in the 3D Viewer represents a 3D model of a scene rendered by combining the depth and color data, as shown in Figure 5.