Float: One-Handed and Touch-Free Target Selection on Smartwatches

We present Float, a wrist-to-finger input approach that enables one-handed and touch-free target selection on smartwatches with high efficiency and precision using only commercially-available built-in sensors. With Float, a user tilts the wrist to point and performs an in-air finger tap to click. We combine the photoplethysmogram (PPG) signal with accelerometer and gyroscope to detect finger taps with a recall of 97.9% and a false discovery rate of 0.4%. Experiments show that using just one hand, Float allows users to acquire targets with size ranging from 2mm to 10mm in less than 2s to 1s, meanwhile achieve much higher accuracy than direct touch.

The Float project was started in 2016, and the related paper was published in 2017. Glad to see that Apple released AssistiveTouch on Apple Watch in 2021, which has exactly the same interaction and technical implementation as Float.

Touch is still the most dominant way users interact with smartwatches, which benefits from its nature of direct input. We however observe two typical problems of using touch on smartwatches. First, touch on smartwatches always requires the “free hand” on which the smartwatch is not attached. Interactions will be inconvenient when the other hand is occupied such as carrying objects, holding an umbrella and riding a bike; The second is that of the fat finger problem, which is exacerbated by the ultra-small display on smartwatches.

Based on these two limitations, we are motivated to seek out alternative input modalities that enable one-handed and non-touch interactions on smartwatches. Our goal was to provide more hands-free interactions, while improving the accuracy and efficiency of target selection beyond direct touch. We propose Float. The basic idea is simple: a user tilts the wrist to point and performs an in-air finger tap to click. For example, a user can handle incoming calls with a single hand while riding a bicycle.

Float is a wrist-to-finger input interaction. This figure shows a walkthrough of the technique and its coexistence mechanism with touch.

• Raise-to-wake: to interact with the smartwatch, the user raises the arm and turns the wrist so that the watch face faces their line of sight. This raise-to-wake action is detected to light up the screen.
• Enabling Float: The user performs the first in-air finger tap as the enabling command to activate the Float interactive mode, after which a (virtual) cursor appears at the center of the watch face.
• Tilt to point: To select the target, the user tilts the wrist towards a certain direction by a certain angle. The direction and angle codetermine the cursor location on the watch face.
• Finger tap to select: Once the correct position is visually verified, a finger tap as the clicking action causes a brief highlight and completes the selection.
• Disabling Float: Any touch event on the screen or lowering the arm disables Float.

Float maps wrist tilt to positions in polar coordinates on the two-dimensional watch face. When the watch face tilts to the current orientation from the initial reference, we regard the tilt direction as the cursor ’s angular coordinate and map the inclination to the radial coordinate.

Suppose \( C_{0} \) and \( C_{1} \) are the initial and the current coordinate system of the watch face respectively. When in \( C_{0} \), the cursor locates at the center of the watch face, \( (0, 0) \) on the xy-plane. The goal is to calculate the cursor’s location after the watch face rotates to \( C_{1} \). Say \( (\alpha, \beta, \gamma) \) is the angle change of the intrinsic rotation around the x, y, z axes respectively which transforms \( C_{0} \) to \( C_{1} \) in the order of z → x → y, and \( R \) is its corresponding rotation matrix. So, the unit outward normal vector \( n_{0} = (0, 0, 1)^T \) of the watch face rotates to:

$$ \begin{eqnarray} n_{1} & = & R \cdot n_{0} \\ & = & \left[ \begin{array}{ccc} cos\beta \cdot cos\gamma + sin\alpha \cdot sin\beta \cdot sin\gamma & -cos\beta \cdot sin\gamma + sin\alpha \cdot sin\beta \cdot cos\gamma & cos\alpha \cdot sin\beta \\ cos\alpha \cdot sin\gamma & cos\alpha \cdot cos\gamma & -sin\alpha \\ -sin\beta \cdot cos\gamma + sin\alpha \cdot cos\beta \cdot sin\gamma & sin\beta \cdot sin\gamma + sin\alpha \cdot cos\beta \cdot cos\gamma & cos\alpha \cdot cos\beta \end{array} \right] \cdot \left[ \begin{array}{ccc} 0 \\ 0 \\ 1 \end{array} \right] \\ & = & (cos\alpha \cdot sin\beta, -sin\alpha, cos\alpha \cdot cos\beta)^T \end{eqnarray} $$

Projecting \( n_{1} \) onto xy-plane of \( C_{0} \) gives the tilt direction:

$$ Dir = atan2(y_1, x_1) $$

The angle between \( n_{1} \) and \( n_{0} \) gives the inclination:

$$ Inc = arccos(z_1) $$

To better understand how users expect to use Float, we conducted a preliminary user study where participants were instructed to explore the appropriate tilt space and elicit confirming actions themselves. Study results were used to determine the final design of Float.

For tilt range determination, overall, the appropriate inclination area of wrist-tilt is nearly an oblique ellipse rather than a circle, with an average of 30.15^o. The upward direction (around 90^o) had the largest inclination range of 39.34^o, while the lower left had the smallest range of 23.91^o. The results suggested that due to the specific wrist structure, participants tended to incline by different angles along different directions. Therefore, to maximize the agreement between users’ tilt-motion intention and the displayed cursor position, we scaled the inclination to different ranges as the tilt direction varies.

For user-defined clicking actions, five interaction modalities were elicited, including dwell-time, voice control, finger tap, wrist motion and head / gaze control. Eleven out of 12 participants voted finger air-tap as their favorite option. Finger tap was recognized as a clear, easy-to-perform and control, and relatively quick method.

The technical implementation of Float includes three parts corresponding to the three transitional actions: raise to wake, tilt and finger tap. Three types of sensors were used: the motion sensor (accelerometer and gyroscope), position sensor (geomagnetic field) and the heart rate monitor (HRM) sensor.

The motion sensor and the position sensor reflect users’ arm and wrist movements, enabling us to recognize the “raise to wake” action and compute the cursor position.

The HRM sensor is located on the rear panel of the smartwatch. It detects the PPG signal which measures the amount back of IR/RED light reflected from the blood vessel in the skin. When a user performs finger gestures, significant patterns can be observed from the PPG signal due to the deformation and strain of the wrist skin. So we used the HRM sensor, combined with the motion sensor, to detect users’ clicking action through finger tap. The HRM sensor has been embedded in most commercial smartwatches.

Accuracy. In stationary context, participants achieved almost 100% of accuracy on all target sizes from 2mm to 10mm with Float. This benefited from its clear visual feedback and participants could directly verify whether the cursor was inside the target or not. In addition, a high pointing accuracy on 2mm (98.9%) suggested that participants could hold the cursor steadily by controlling the wrist. During walking, there was an obvious decrease on performance on size 2mm and 3mm with an accuracy of 71.5% and 91.5%, respectively. With size no less than 4mm, however, participants were able to perform high-precision (>98.0%) target selection with Float. It is also demonstrated that Float was more accurate than direction touch over all target sizes, especially for smaller targets from 2mm to 6mm.

Completion Time. The completion time of Direct Touch was significantly smaller than Float, which benefited from its nature of direct input. For Float, there was a significant effect of Target Size on completion time.