ShoeSense

When the user is engaged with a real-world task it can be inappropriate or difficult to use a smartphone. To address this concern, we developed ShoeSense, a wearable system consisting in part of a shoe-mounted depth sensor pointing upward at the wearer. ShoeSense recognizes relaxed and discreet as well as large and demonstrative hand gestures. In particular, we designed three gesture sets (Triangle, Radial,and Finger-Count) for this setup, which can be performed without visual attention. The advantages of ShoeSense are illustrated in five scenarios: (1) quickly performing frequent operations without reaching for the phone, (2) discreetly performing operations without disturbing others, (3) enhancing operations on mobile devices, (4) supporting accessibility, and (5) artistic performances.
We present a proof-of-concept, wearable implementation based on a depth camera and report on a lab study comparing social acceptability, physical and mental demand, and user preference. A second study demonstrates a 94-99% recognition rate of our recognizers.

Left: ShoeSense perspective (image from S. Malacria). Right: Triangle gesture.
Triangle gesture set. Apex of each triangle is the wrists (1,3), hands (2), and arms (4,5) respectively.
3D radial gesture set (Left, Right, Front, Back, Up, Down) mapped to frequent actions on a smart phone.
Finger-Count gesture set. Pinch on one hand (registration) and extended fingers on the other hand.
The ShoeSense prototype. The system is wholly contained as a wearable computer. A novel version has now been proposed. See above.
(a) A user performing a hand-to-left-wrist triangle; (b) a 3D radial gesture; (c) a 5 finger-count gesture.
3D control on mobile devices. (a) Moving the finger along the arm (modifying the shape of the triangle) rotates the 3D object. (b) x, y and z translation with 3D radial gestures. (c) The user chooses a predefined control with a finger-count gesture (3 fingers on the right hand + a pinch formed by the left hand and the device). (d) Finger-count gestures to select commands on a watch (simulated by an iPhone on the wrist).
Social acceptability by location and gesture set (10-point scale, 10 is the best). 95% confidence interval marked
Social acceptability by audience and gesture set (10-point scale, 10 is the best). 95% confidence interval marked.