Body Part Recognition and the Development of Kinect

Late 2010, Microsoft launched Xbox Kinect, an amazing new depth-sensing camera and a revolution in gaming where your body movements allow you to control the game. In this talk I'll present a behind-the-scenes look at the development of Kinect, focusing on the depth camera, the challenges of human pose estimation, and the body part recognition algorithm that drives Kinect's skeletal tracking pipeline. Body part recognition uses machine learning to efficiently produce an interpretation of pixels coming from the Kinect camera into different parts of the body: head, left hand, right knee, etc. The approach was designed to be robust: firstly, the system is trained with a vast and highly varied training set of synthetic images to ensure the system works for all ages, body shapes & sizes, clothing and hair styles; and secondly, the recognition does not rely on any temporal information, allowing the system to initialize from arbitrary poses and preventing catastrophic loss of track.