Humans As Robot

Human intelligence relies on a wealth of commonsense acquired from a lifetime of experience. In order to achieve the long term goals of artificial human intelligence, researchers must find ways to endow machines with this type of commonsense. Humanoid robots could serve as a direct approach to the acquisition of this type of competence, since a sufficiently sophisticated humanoid robot would be able to experience much of the world in the same way as humans. Currently, however, humanoid robots have very limited experience with the world due to obstacles ranging from mechanical design to social constraints on the use of autonomous robots. Wearable computing systems have the potential to measure a great deal of the sensory input and physical output of a person as he or she experiences everyday activities. Much can be learned through passive observation of these measurements. However, if we can also find ways for the wearable system to strongly influence the behavior of the person wearing the system, many learning tasks can be made easier. We are designing Duo, a wearable creature that works with a cooperative human in order to learn about everyday objects in the world and the ways in which they are commonly used. The wearable learns about the world by watching and sometimes making requests of the wearer as he goes through activities in the day. By using the same sensory input as the person and co-opting his output behaviors, the wearable creature serves as a top layer of control in a subsumption architecture with the human serving as a powerful mechanical and computational infrastructure. As an initial exploration into this class of wearable applications, we are creating a wearable system that attempts to learn common-sense about everyday actions as they relate to objects and changes to the environment. As shown in the diagram at the top of the page, the system currently consists of a hat-mounted camera from which the creature watches the world and 4 Intersense devices, each of which provides an absolute orientation, with which the creature measures the kinematic configuration of the wearer's head and dominant arm. The creature also serves as a high level controller that attempts to co-opt the wearer's behaviors by requesting actions through headphones. For example, the creature can currently request that the wearer look at an object that the wearer is manipulating, in order to see it better and segment it using the LED array. Likewise Duo can ask that the wearer keep his head still, in order to make perception easier. We hypothesize that a broad array of actions useful for learning can be successfully prompted by speech from Duo. In the future Duo may ask the wearer to repeat an action by uttering, ``do that again!'', which should help the creature segment the activity into meaningful parts. More generally, by requesting actions the wearable creature can test hypotheses it has made about actions and their effect in the world. Duo's first behaviors work together with a cooperative human to acquire high-quality segmentations of everyday manipulable objects used by the person wearing the system. When Duo detects that the wearer has reached for an object it asks the wearer to look at it with speech through the headphones. While looking at the object Duo flashes the LED array in order to segment the hand and object, which are in the foreground, from the rest of the world in the background. While looking at the object, Duo also monitors the wearer's head movements. If the wearer's head moves significantly, Duo request that the wearer keep his head still. We are currently working with these segmented images to perform object tracking, detection, and recognition. We are also creating segmentation and learning code for the object related actions of the wearer. Our overall goal is to develop a viable system for the acquisition of commonsense related to everyday human activities. A successful creature would be able to learn and control a set of common behaviors performed by a cooperative human and would be able to relate common action patterns to the visual appearance of objects and to observed changes in the world.