Researchers at HIIT and Max Planck Institute for Informatics show how computer vision -based hand tracking and vibration feedback on the user's hand can be used to steer the user's hand toward an object of interest. A study shows an almost three-fold advantage in finding objects from complex visual scenes, such as library or supermarket shelves.

Finding an object from a complex real-world scene is a common yet time-consuming and frustrating chore. What makes this task complex is that humans' pattern recognition capability reduces to a serial one-by-one search when the items resemble each other.

Researchers from the Helsinki Institute for Information Technology HIIT and the Max Planck Institute for Informatics have developed a prototype of a glove that uses vibration feedback on the hand to guide the user's hand towards a predetermined target in 3D space. The glove could help users in daily visual search tasks in supermarkets, parking lots, warehouses, libraries etc.

The main researcher, Ville Lehtinen of HIIT, explains "the advantage of steering a hand with tactile cues is that the user can easily interpret them in relation to the current field of view where the visual search is operating. This provides a very intuitive experience, like the hand being 'pulled' toward the target."

The solution builds on inexpensive off-the-shelf components such as four vibrotactile actuators on a simple glove and a Microsoft Kinect sensor for tracking the user's hand. The researchers published a dynamic guidance algorithm that calculates effective actuation patterns based on distance and direction to the target.

In a controlled experiment, the complexity of the visual search task was increased by adding distractors to a scene. "In search tasks where there were hundreds of candidates but only one correct target, users wearing the glove were consistently faster, with up to three times faster performance than without the glove", says Dr. Antti Oulasvirta from Max Planck Institute for Informatics.

Dr. Petteri Nurmi from HIIT adds: "This level of improvement in search performance justifies several practical applications. For instance, warehouse workers could have gloves that guide them to target shelfs, or a pedestrian could navigate using this glove. With the relatively inexpensive components and the dynamic guidance algorithm, others can easily build their own personal guidance systems."