Beyond remote control: Exploring natural gesture inputs for smart TV systems

Abstract

This paper reports on work that explores natural gesture inputs from end-users for television use in a typical living room setting. First, we derive a set of 19 user-defined freehand gestures for regular TV control tasks. This study helps us to determine the gestures preferred by TV viewers and reduce the risk of developing an unusable, ineffective system. Then, based on this user-defined gesture set, we propose a unified framework to address specific problems in a complex real-world TV viewing environment, including 1) the automatic exclusion of many meaningless daily actions by TV viewers, 2) the capability to recognize both static and dynamic gestures simultaneously, as well as one- and two-handed gestures simultaneously, and 3) the continuous recognition of multiple dynamic gestures in the air (e.g., a channel switching gesture for channel 127). Experimental results show that our approach allows users to interact with TV-based applications more flexibly and effectively, with improved user experience and user satisfaction. Finally, we highlight the implications of our work for the design and development of related freehand gesture applications.

Keywords

Smart home human–computer interaction user-defined gestures gesture recognition coupled hidden Markov model television remote control

1. Introduction

With 27 degrees of freedom (DOF), human hands are used to perform various tasks, simple or complex, in real life. Rapid technical developments in areas such as natural human–computer interaction (HCI), sensor technologies (e.g., time-of-flight camera, Kinect and Leap Motion), and motion capture techniques have brought freehand-gesture-based interaction into application domains, including virtual/augmented reality [7], mobile computing [31,36], children’s games [13], ubiquitous computing [45], and intelligent robotics [14]. Freehand refers to human beings’ natural hands. Freehand-gesture-based interaction is a method of interaction using only bare hands for HCI input; it can potentially deliver natural, intuitive, and terse but powerful human–computer interaction techniques.

In addition to PC-based applications, the smart home is one area where freehand-gesture-based interaction becomes important. When our homes are equipped with an increasing number of computational devices, methods to support additional natural user interaction with these devices becomes a challenge. Currently, network-based applications and on-demand TV programs are delivered directly to an interactive digital TV. Compared with traditional TV tasks, such as channel switching, interactive tasks in a digital TV, such as entering characters to search contents and navigating through content space, are more complex and time-consuming. Specifically, using a traditional remote control often leads to low input efficiency and usability problems, such as eye-hand separation, i.e., users need to shift their attention back and forth between the TV screen and the remote control.

To support more natural user interaction with TV-based applications and contents, freehand-gesture-based designs have been explored [8,18,21,32,33,39,45]. Most of these designs can help users focus their eyes on the TV screen while performing various command gestures in the air. While freehand gesture inputs can provide increased variety and flexibility, current approaches continue to face challenges, such as robustness of gesture recognition, capacity to address different types of gestures, continuous recognition of multiple dynamic gestures in the air, and usability of freehand-gesture-based interaction techniques in practice. These challenges have limited the broad application of freehand-gesture-based designs in TV-based applications.

To address some of these challenges, we created a living room environment and then asked users to interact with a television using various gestures. Compared with existing gestural TV systems [8,21,32] that do not allow users to create their own gestures, we propose allowing users to create their own user-friendly gestures that will maximize their comfort and ease of use. Participatory design has long been considered to be an effective method for understanding end-users’ mental models and behaviors with new interaction techniques at the early stage of system design. With this method, system designers can obtain valuable information to shape a product’s characteristics for more effective and efficient use. Gesture elicitation studies, which is a technique that has emerged from the field of participatory design, have attracted increasing attention and been widely used to collect end-users’ requirements and expectations regarding the target system by involving end-users in gesture design processes. Based on the results of a gesture elicitation study, we propose a framework for recognition of the whole set of user-defined gestures. We highlight the robustness and high recognition accuracy of our framework for continuous recognition of different types of gestures.

The remainder of this paper is structured as follows: Section 2 reviews related work. Section 3 presents the system requirements and the user-defined gesture set. Section 4 proposes a unified framework for the recognition of user-defined gestures for interactive digital TV and describes some key technologies in the framework. Section 5 describes two experiments and a comparative analysis to study the performance of freehand gestures and presents the experimental results. Section 6 discusses the contributions of our research, its limitations, and possible future research directions.

2. Related work

Our work primarily concerns research related to the participatory design of freehand gestures and development of gesture-based prototypes for HCI. These two aspects of our research will be considered through the literature review.

2.1. Freehand-gesture-based systems

Existing work on freehand gesture recognition can be roughly divided into two classes – static gesture recognition [7,16,30] and dynamic gesture recognition [17,20,40–42,44]. A static gesture only concerns static features, such as hand shape, size, and color, while a dynamic gesture involves dynamic spatiotemporal information, such as movement speed, direction, position, and duration of the gesture. By using computer vision and HCI techniques, both static and dynamic gestures can be mapped directly to different computer commands.

Freehand gestures have been used in various ways. Some applications simply used gestures as a “natural mouse” for such tasks as pointing and drawing [24], while other applications used gestures for more complex activities, such as navigation and object manipulation in a virtual environment [7,16] and on interactive surfaces [12].

Beyond PC-based applications, freehand gestures are also widely used in smart homes for controlling televisions, air-conditioners, etc. For example, Freeman et al. [8] created a prototype to replace a TV remote control with freehand gestures. However, this system was developed based on the window, icon, menu, pointing device (WIMP) paradigm, and still treated freehand gestures as mouse commands. The rapid development of depth sensing technologies [11,29] in recent years allows users to move beyond the traditional WIMP paradigm and desktop metaphor and to involve users in a more flexible, creative and intuitive way regarding TV-based applications by translating freehand gestures directly into interaction commands. For example, Zaiţi et al. [45], Takahashi et al. [32], and Lee et al. [21] explored the use of freehand gestures to interact with a TV set by using a time-of-flight (TOF) camera, Kinect, and Leap Motion, respectively.

However, the gestures in those systems were designed not by end-users, but by experts with significant technical experience in computer vision, image processing, and HCI. End-users have few opportunities to participate in gesture design. Such practices may lead to a disagreement between user gestures imagined by designers and actual user gestures. Similar to the vocabulary problem [9] that affects the performance of information retrieval systems, this gesture disagreement problem may lead to a risk of poor system usability and low user acceptance. To develop a user-friendly gesture system, we must carefully evaluate the types of gestures that are most commonly used and structure our design principles accordingly.

2.2. Participatory design of freehand gestures

To gain a better understanding of these issues, some researchers used participatory design methods to study user behaviors. For example, the Wizard of Oz (WOZ) method was used by Höysniemi et al. [13] to study favorite gestures for small children playing computer games. Wobbrock et al. [37] developed a user-defined gesture set for surface computing based on the results of a guessability study. Such participatory design approaches have also been used for mobile interaction [31], music control at home [23], and TV control in a smart room [33,34,45]. Research has provided empirical evidence on the benefits of practices involving end-users in gesture design. For example, by comparing user-authored surface gestures and researcher-authored surface gestures, Morris et al. [26] found that researcher-authored gestures are less memorable and discoverable than user-authored gestures.

Different from those pure participatory design methods mentioned above, another approach for freehand gesture design is to involve end-users in multiple steps of the gesture design process. For example, Löcken et al. [23] and Nielsen et al. [27] proposed eliciting multiple candidate gesture sets from end-users, rather than only one in the early stage of gesture design. Then, a benchmark test was used in the subsequent process to compare and validate the usability of different gesture candidates and refine the results. Under this approach, additional feedback from end-users can be collected and the risk of rejecting promising gesture candidates can be reduced.

However, these studies generally stopped at the stage of gesture design and definition and lack further validation of gesture recognition and user preferences in practice. To improve the research in this direction, we conducted a study of natural gesture inputs for TV-based interaction tasks. The aim of this study is to explore various problems in the process of design, implementation, and evaluation of freehand gesture systems, specifically those concerning such usability problems as

What are the most commonly used gesture techniques for TV commands such as volume control and channel changing?

What is the best way to design and develop such gesture techniques for TV systems?

Can those gestures can work effectively in real-world scenarios?

Do users prefer those gestures for TV-based applications?

3. Requirement analysis and user-defined gestures

To develop a user-friendly gesture application, it is necessary to have a clear understanding of the usage context of the intended system. However, most existing research of gesture design and development focuses on concrete gesture recognition algorithms; few are concerned with gestural design principles and design specifications in practice. Compared with prior work [8,18,21,32,45], we conducted a multi-stage participatory design study in which actual TV viewers were asked to work with expert designers to design gestures for a television system.

During the first stage, we recruited 24 participants (10 males and 14 females) from a university. Their ages were between 18 and 45 ( $M = 30$ , $SD = 4.62$ ). Participants had different educational backgrounds, including art design, computer science, history, and psychology. Participants were asked to brainstorm functional requirements and interactive context from their own perspectives. None had previous experience in interaction design, nor had they previously used freehand gesture interaction techniques (such as Kinect and Leap Motion). As a result, we derived 19 most needed functions for interactive digital TV (Table 1).

Table 1
User-defined gesture set for a smart TV system

Function Gesture Initial hand posture Following hand motion trajectory

Previous elicitation studies [23,27,31,33,34,37,45] required participants to design only a single gesture for each target task. However, these methods often suffer from the legacy bias problem [25], which refers to the phenomenon that end-users’ gesture proposals are often biased by their experience with prior interfaces and technologies such as the WIMP interfaces (e.g., a mouse) or touch-based interfaces (e.g., an iPhone). To reduce the impact of legacy bias, we conducted an improved gesture elicitation study inspired by Morris’ suggestion [25] during the second stage. During this stage, the same 24 participants were asked to design 3 gestures rather than one for each of the 19 commonly seen TV tasks for daily use based on their preferences. By doing this, we hope to derive more reliable gesture set for TV control than traditional gesture elicitation studies.

Before the study, we established an environment that mimicked a living room in a usability lab. Then, participants sat on a sofa and were shown a list of all 19 tasks on the TV screen. We used the “think-aloud” method to gather the meaning of participants’ gestures. All participants were told that they were interacting directly with the TV using a gesture recognition system. They were asked to speak loudly about what tasks they were performing. We adopted a Wizard of Oz approach in responding to participants’ gestures. An experimenter listened to what a participant said when performing a task and used a regular TV remote control to produce the result of the task. We used five cameras to capture participants’ intuitive gestures from different perspectives for later data analysis. As a result, we collected a total of 1368 gesture candidates ( $19 \times 24 \times 3$ ) for these 19 TV tasks.

Before the third stage, we conducted a brainstorming session in which 4 HCI researchers were asked to merge and group gesture candidates for each TV function. For gestures with the exact same hand shape and/or same motion trajectory, researchers grouped them into a single gesture directly. For gestures with similar characteristics in hand shape and/or motion trajectory, researchers replayed the corresponding video files we captured during the elicitation process and discussed whether and how to group gestures based on participants’ verbal explanations. For example, 19 swipe right gestures with different hand shapes for “Next Channel” can be merged into a single gesture. As a result, we obtained 252 groups of identical gestures for 19 TV tasks. During the third stage, we recruited another 24 participants (12 males and 12 females) with an average age of 31.34 years ( $SD = 8.79$ ). Their occupations included restaurant host, housewife, public affairs consultant, students, and professors. Then, the 252 groups of identical gestures were evaluated and refined by the 24 participants on a 7-point Likert scale in terms of matching degree, i.e., to what extent end-users can match a gesture to the corresponding TV function; memorability, i.e., how easy it is to remember a specific gesture for a given TV function; and comfort, i.e., how easy it is to perform a gesture physically. Based on the experimental results, we selected an optimal gesture (i.e., the gesture candidate with the highest average score) for each task and subsequently generated a collection of 19 user-defined gestures for the desired 19 common TV commands. (Table 1).

As shown in Table 1, the user-defined gesture set involves three static gestures (Confirm, Mute, and Menu) and 16 complex dynamic gestures. Compared with simple dynamic gestures that only concern hand motion trajectories [17,20,42–44], the complex dynamic gestures in our work involve both an initial presence of a static hand posture and the following spatiotemporal trajectories of hand movement, i.e., complex dynamic gestures assume the presence of a sequence of hold-movement-hold. For example, the dynamic gesture ‘0’ is recognized as a combination of an initial ‘index finger pointing up’ hand posture and a following dynamic gesture to draw a circular trajectory with the index finger. In addition, the user-defined gesture set involves both one- and two-handed gestures simultaneously. The three two-handed gestures are Mute, Turning on the TV, and Turning off the TV; the remainder are one-handed gestures.

In this section, end-users were asked to design gestures freely according to their preferences. The user-defined gesture set established the foundation for the remaining system development process.

4. The proposed gesture-based television system

From the participatory design study, we learned what gestures end-users prefer and a gestural TV system should provide. To recognize the proposed 19 user-defined gestures for TV control tasks in a natural human-machine environment, our system must meet the following requirements:

Recognize both static and complex dynamic gestures simultaneously;

Recognize both one- and two-handed gestures simultaneously;

Distinguish between the users’ meaningful gestures and their meaningless actions, e.g., a meaningful Previous Channel gesture and a meaningless waving hand action; and

Recognize multiple continuous dynamic gestures in the air, e.g., a channel switching gesture for channel 127.

Fig. 1.

System architecture.

Algorithm 1

The mean shift procedure

Based on the above requirements, we propose a unified freehand gesture recognition framework. The proposed framework, as shown in Fig. 1, consists of three main functional modules: hand segmentation, feature extraction, and gesture recognition; each addresses a specific problem. The hand segmentation module receives live video feeds from a contactless depth sensor and sends segmented binary hand images to the feature extraction module. The image vision feature is extracted and represented by using density distribution features and vector quantization techniques in the feature extraction module; it is then processed by a gesture recognition module. For dynamic gesture recognition, the system must execute hand posture and hand motion trajectory recognition modules simultaneously, and switches from hand posture to hand motion trajectory recognition by using a simple motion activated mechanism that uses two cutoff thresholds in a twin-comparison strategy [46]. To identify key gestures and reject meaningless actions, we propose a new filter model and incorporate it into the gesture recognition module, which in turn can recognize both one- and two-handed gestures simultaneously. This helps to classify static and dynamic gestures in a unified framework. We also provide a mapping table for users to define their personal mappings between gestures and TV controlling commands.

4.1. Hand segmentation

Mean shift is a robust non-parametric clustering technique that has been widely used for image segmentation and feature space analysis [5]. Because it is immune to prior knowledge of the number of clusters, we decided to use a mean shift procedure to segment human hand(s) from other complex objects in a complex TV viewing environment. The algorithm translates the kernel window with the profile $k_{N} (x)$ along the gradient direction of the probability density function of the depth image. Given an initial point x and an allowable error ε, the mean shift procedure is given as follows in Algorithm 1.

After the mean shift segmentation, some noise may remain in the binary image. The fuzzy set theory and pyramid method in morphology [47] are introduced to filter and enhance the binary image. Next, a circle fitting procedure [28] is used to detect the palm by estimating the largest fitting circle in the binary image (Fig. 2b). Similar to the mean shift algorithm, the circle fitting procedure is iteratively repeated until it isolates the palm from the remainder of the forearm (Fig. 2c).

Fig. 2.

Different stages of the hand segmentation process: a) a binary image containing a roughly segmented hand by running the mean shift segmentation procedure; b) palm detection by running the circle fitting procedure; c) refined hand region; and d) an equidistance division of the hand region.

4.2. Feature extraction and description

4.2.1. Static feature extraction

As shown in Fig. 2, the hand shape is represented by white pixels distributed in different spatial regions of a binary image. Therefore, we can judge whether two binary images are similar by seeing if they have a similar regional distribution of white pixels. In this section, we propose a density distribution feature (DDF) method to describe the hand region in a binary image. Given a binary image, the DDF of the hand region is defined as follows: $\begin{matrix} (1) & DDF = (r_{1}, \dots, r_{N}; d r_{1}, \dots, d r_{N}) \end{matrix}$ where r and $d r$ are two N-dimensional feature vectors; r refers to the relative density of the white pixels in each small region produced by separating the hand binary image equally; and $d r$ defines the one-order difference of r in the polar coordinate system. Then, the DDF of the hand region is calculated with the following steps in Algorithm 2.

Algorithm 2

The calculation of DDF

As shown above, the DDF of a hand region is invariant to translation, rotation, and scaling.

4.2.2. Dynamic feature extraction

Conventional approaches for dynamic gesture recognition typically rely on 2D hand motion trajectories [20,28,43,44]. Due to the loss of feature information, systems developed using these methods suffer from the degradation of recognition accuracy. Compared to prior work, we directly extract 3D feature points from the input frames. Then, in the tracking phase, we connect all the centroid points of the hand region to produce a 3D hand motion trajectory.

Let $P_{t} = (x, y, z)$ and $P_{t + 1} = (x^{'}, y^{'}, z^{'})$ be two adjacent points in a motion trajectory. Their difference can be represented as a vector $\vec{v}$ : $\begin{matrix} (2) & \vec{v} = P_{t + 1} - P_{t} = {[Δ x, Δ y, Δ z]}^{T} \end{matrix}$ where $Δ x$ , $Δ y$ , and $Δ z$ are three spatial components of the difference vector. To meet the requirement of invariance to translation, rotation, and scaling, the difference vector is normalized using the following equation: $\begin{matrix} (3) & \frac{\vec{v}}{| v |} = {[Δ x^{'}, Δ y^{'}, Δ z^{'}]}^{T} \end{matrix}$

Based on the proposed procedure, a 3D motion trajectory is characterized by a feature vector set $V = {v_{i}, i = 1, 2, \dots, N}$ . Next, an extended Linde–Buzo–Gray (LBG) algorithm [22] based on the fuzzy clustering theory is used for vector quantization. Given the unlabeled feature vector set $V = {v_{i}, i = 1, 2, \dots, N}$ , our aim is to divide V into n fuzzy subsets ( $1 < n < N$ ). The extended LBG algorithm is given as follows in Algorithm 3.

Algorithm 3

The extended LBG algorithm

We used the 16 complex dynamic gestures (Table 1) to generate codebooks of sizes $S = 2^{q}$ ( $q = 1, \dots, 10$ ) and measured the computational time needed by the CPU in seconds per iteration. We ran our algorithm using a computer with a dual-core CPU at 2.4 GHz. Experimental results show that the computational time exhibits an exponential growth when q is greater than 5. However, our algorithm provides superior performance for vector quantization when q is 5 rather than 1, 2, 3, and 4. Therefore, we set the size of the codebook to $S = 2^{5} = 32$ . Consequently, each feature vector $v_{i}$ in V is converted to one of 32 symbols (i.e., $y_{i}$ ) in $A_{t + 1}$ and then input into the gesture CHMMs for classification.

4.3. Gesture recognition and classification

4.3.1. Hand posture recognition

Static gesture recognition is often based on template matching-based methods [16,30]. In this work, we adopted the Euclidean distance method to find the similarity between samples and template images. It should be noted that the two N-dimensional feature vectors r and $d r$ belong to different feature spaces, so they must be normalized to the 0-1 range to form a new global distance before being used for comparison.

We performed the normalization procedure using the Gaussian model method. Given an input gesture sample $S = (r_{1}^{S}, \dots, r_{N}^{S}; d r_{1}^{S}, \dots, d r_{N}^{S})$ and the target class template $T = (r_{1}^{T}, \dots, r_{N}^{T}; d r_{1}^{T}, \dots, d r_{N}^{T})$ , the classification procedure is given as follows in Algorithm 4.

Algorithm 4

The classification algorithm

For two-handed gestures, the system should compute the global distances $d_{DDF}^{LH}$ and $d_{DDF}^{RH}$ for the left hand and the right hand, respectively, then find a two-handed class template $T^{* *}$ that yields the minimum global distance: $\begin{matrix} T^{* *} = arg min [d_{DDF}^{LH} + d_{DDF}^{RH}] \end{matrix}$

4.3.2. Dynamic gesture model

The hidden Markov model (HMM) is a popular statistical modeling method used in single hand trajectory recognition in recent years [11,29]. However, the conventional HMM has less effective modeling performance for two-handed gestures because it makes restrictive assumptions that the system generates a single process having a small number of states and an extremely limited state memory. Therefore, the two-handed input signals fail to satisfy the very restrictive Markov condition. To improve this, many new models were proposed, for example, Factorial HMM (FHMM), Layered HMM (LHMM), and Coupled HMM (CHMM). Because of the strong capability of modeling and classifying two coupled random processes (e.g., Chinese martial art and Tai Chi) [1,2,19], the CHMM is used in this paper for two-handed dynamic gesture modeling and recognition. Figure 3 shows examples of the HMM and CHMM topologies.

Fig. 3.

a) HMM topology; b) a two-chain CHMM topology.

As shown in Fig. 3, a CHMM has two chains – a left-hand chain and a right-hand chain. Some parameters for defining the CHMM are given as follows:

An observation vector, $O = {O^{L}, O^{R}}$ , where $O^{L} = {O_{1}^{L}, O_{2}^{L}, \dots, O_{n}^{L}}$ and $O^{R} = {O_{1}^{R}, O_{2}^{R}, \dots, O_{n}^{R}}$ are the observed value of the left-hand and right-hand chains, respectively.

A sequence of joint states, $S_{i j} = {S_{i}^{L}, S_{j}^{R}}$ , where $S_{i}^{L} = {S_{1}^{L}, S_{2}^{L}, \dots, S_{n}^{L}}$ and $S_{j}^{R} = {S_{1}^{R}, S_{2}^{R}, \dots, S_{n}^{R}}$ are the state sequence of the left-hand and right-hand chains, respectively.

Transition probability $P (S_{j k} | S_{i l}) = Ψ (P (S_{j}^{L} | S_{i}^{L}) \cdot P (S_{k}^{R} | S_{l}^{R}), P (S_{k}^{R} | S_{i}^{L}) \cdot P (S_{j}^{L} | S_{l}^{R}))$ , where Ψ is a combining function and can be factored back into the transition probability of the left-hand and right-hand chains.

Observation probability $P (S_{i j}) = P (O_{i}^{L} | S_{i}^{L}) \cdot P (O_{j}^{R} | S_{j}^{R})$ , where $P (O_{i}^{L} | S_{i}^{L})$ and $P (O_{j}^{L} | S_{j}^{L})$ are the observation probability of the left-hand and right-hand chains, respectively.

$P (S_{1}^{L})$ and $P (S_{1}^{R})$ are the prior probability of the left-hand and right-hand chains, respectively.

Based on the above statement, the likelihood function of a CHMM is defined as follows: $\begin{matrix} (4) & \begin{matrix} L (θ) & = P (S_{1}^{L}) \cdot P (S_{1}^{R}) \prod_{t = 1} P (O_{i}^{L} | S_{i}^{L}) \\ \cdot P (O_{i}^{R} | S_{i}^{R}) \cdot P (S_{i + 1}^{L} | S_{i}^{L}, S_{i}^{R}) \\ \cdot P (S_{i + 1}^{R} | S_{i}^{L}, S_{i}^{R}) \end{matrix} \end{matrix}$

Then, the CHMM parameter is estimated based on the maximum posterior probability (MAP) method. In the recognition phase, the Viterbi algorithm [35] is used to decode all gesture CHMM. Given an unknown gesture input, our recognizer attempts to identify the index of the gesture CHMM that produces the maximum probability of the observation symbol sequence and assigns it to the unknown gesture as its class label. To find the best state sequence for a given observation vector O, we need to define the quantity: $\begin{matrix} (5) & \begin{matrix} θ_{MAP} & = arg max_{θ} P (θ | O) \\ = arg max_{θ} \frac{P (O | θ) P (θ)}{P (O)} \\ = arg max_{θ} {\lg [P (O | θ)] + \lg [P (θ)]} \end{matrix} \end{matrix}$ where θ denotes the CHMM parameter, $P (θ | O)$ is the posterior probability of observation vector O for parameter θ, $lg [P (O | θ)]$ is the likelihood function that can be calculated by running the Baum-Welch algorithm [35], and $P (θ)$ is the Dirichlet distribution [10] of parameter θ.

4.3.3. Filter model

A robust gesture recognition system should be able to support multiple continuous gestures and avoid the “Midas Touch problem”, which means each user action may be captured by a camera and then identified as a gesture by the system if there is no effective filtering mechanism in place. In this work, we proposed a visual-attention based filter model to reject meaningless actions not defined in the gesture vocabulary. Compared with prior techniques that rely only on low-level velocity and acceleration information [1], the proposed model introduces context information for meaningful gesture recognition.

Suppose G is an intended meaningful gesture, T is the gesture class to which G belongs, and $L = (L_{s}, L_{e})$ refers to the start and end point of gesture G. Then, given the context information E, the conditional probability of gesture G is calculated as follows: $\begin{matrix} (6) & P (G | E) = P (L | T, E) P (T | E) \end{matrix}$ where the likelihood function $P (T | E)$ corresponds to the visual pre-attention stage in gesture recognition and is calculated based on a training process. The output of $P (T | E)$ can serve as an adaptive confidence measure for gesture spotting (i.e., determining the start and end points of a gesture in continuous video streams).

It is difficult to construct such a filter model, especially in a real TV viewing environment, because there are infinite meaningless actions by the users. In this paper, all meaningless actions are modeled by a unified CHMM where the joint states of all gesture CHMMs are copied with their own self-transition probabilities $S_{i i}$ , and then connected by two new dummy states, named Start State ( $ST$ ) and End State ( $ET$ ). Next, the observation probability of the filter model is re-estimated with a Gaussian distribution smoothing filter so that it can represent any pattern, as follows: $\begin{matrix} (7) & \vec{P (S_{i j})} = \frac{1}{\sqrt{2 π σ}} \cdot exp (\frac{{(P (S_{i j}))}^{2}}{2 σ^{2}}) \end{matrix}$ where $P (S_{i j})$ is the observation probability of joint state $S_{i j}$ in the original gesture CHMM.

Then, the transition probability between two non-dummy joint states $S_{i l}$ and $S_{j k}$ in the filter model is defined as: $\begin{matrix} (8) & \vec{P (S_{j k} | S_{i l})} = \frac{1 - P (S_{j k} | S_{i l})}{N - 1} \end{matrix}$ where N is the total number of joint states in all gesture CHMMs and $P (S_{j k} | S_{i l})$ is the transition probability from joint state $S_{i l}$ to $S_{j k}$ in the original gesture CHMM.

Because these two dummy states ( $ST$ and $ET$ ) do not observe symbols and are passed quickly without time delay, each state in the filter model can be reached from any other state in a single transition. In this way, the filter model can be used to model any meaningless actions in practice.

Fig. 4.

Gesture spotting framework.

4.3.4. Gesture spotting and recognition

After the construction of the filter model for all one- and two-handed gestures in our user-defined gesture set, a relative entropy method [20] is used to reduce the states in the filter model and improve system performance. Finally, two filter models are embedded into the gesture spotting framework – one filters meaningless actions before a meaningful gesture and the other filters meaningless actions after a gesture (Fig. 4).

Then, we use a sliding window technique to calculate the observation probability of the gesture HMMs $P (O | λ_{g})$ and the filter model $P (O | λ_{filter})$ .

Let $Δ = P (O | λ_{g}) - P (O | λ_{filter})$ . If Δ changes from negative to positive, we can assume that the system detects the start point of a gesture. Otherwise, if Δ changes from positive to negative, the end point of the gesture is detected and then the system will activate the gesture recognition model. Figure 5 shows the workflow of the sliding window.

Figure 6 presents an example of the temporal evolution of the filter model and multiple continuous dynamic gestures of “Channel 127” in the process of recognition. In the first 21 frames, the probability of the filter model is the greatest. Dynamic gesture 1 is then detected at frame 21 and has the highest probability thereafter. After 20 frames, the probability of dynamic gesture 1 drops nearly to zero and the probability of the filter model rises to the highest. Next, the system successfully detects dynamic gestures 2 and 7 at frames 65 and 105, respectively. From Fig. 6, we can see that the filter model provides a confidence measure of whether to accept the input as a gesture or reject it as a meaningless action.

Fig. 5.

Workflow of the sliding window.

Fig. 6.

The likelihood evolution of the gesture CHMMs and the filter model for the continuous recognition of gesture Channel 127.

5. Experimental study

We created a living room environment and then conducted two experiments and a comparative analysis to study the performance of our user-defined freehand gestures in TV-based applications. The purpose of our research was to investigate the following problems:

What is the performance of the proposed gesture recognition system? Is it good enough to identify the user-defined gestures in practice?

How does freehand-gesture-based input perform in comparison with the conventional remote control in TV-based applications that require complex navigation and item selection? How will people accept freehand-gesture-based input?

5.1. Experiment 1: Performance evaluation of our gesture recognition system

The main purpose of Experiment 1 was to evaluate the recognition accuracy of our gesture recognition system.

5.1.1. Participants and apparatus

In this experiment, we recruited 20 participants, 10 males and 10 females, from a university. Their ages were between 18 and 45 ( $M = 26$ , $SD = 7.9$ ).

We created a living room environment equipped with a 42-inch SONY TV, a depth camera, a remote control, and a PC with a dual-core 2.4 GHz CPU and 4G memory. The TV and the depth camera were connected to the PC, which hosted our system to recognize the users’ gestures and deliver contents to the TV.

5.1.2. Procedure

Each participant performed 19 gesture samples, including 3 static gestures and 16 dynamic gestures. Each gesture was performed ten times by 20 participants. The sample collection procedure lasted approximately 2 to 3 hours for each participant.

5.1.3. Results

In total, 3,800 gesture samples ( $19 \times 10 \times 20$ ) were collected. Using a leave-one-participant-out cross-validation, the test results are illustrated by the confusion matrices in Tables 2 and 3.

Table 2
The confusion matrix of static gesture recognition

Table 2 shows that the mean recognition accuracy of the three static gestures is 98.5%. The two most confused gestures are Confirm and Menu. This may be due to the similarity of the shape of the two gestures, or the inter-user differences in performing these two gestures.

As described above, each dynamic gesture in this study can be decomposed into an initial hand posture and following hand motion trajectories. Considering the possible influence that the initial hand posture could exert on the recognition of the subsequent hand motion trajectories, we compared the results of the 16 dynamic gestures with and without initial hand postures. For dynamic gestures without initial hand gestures, we only compared their motion trajectories. The experimental results are shown in Table 3.

Table 3

The confusion matrix of dynamic gesture recognition with initial hand-pose (upper values) and without initial hand-pose (lower values)

Table 3 shows that the mean recognition accuracy of the 16 dynamic gestures without initial hand postures and with initial hand postures are 94.3% and 92.2%, respectively. Lower performance in recognition of dynamic gesture with initial hand posture is probably because both hand posture and hand motion trajectory recognition modules are concurrently executed. As a result, ambiguities might arise in the misidentification or un-identification of an initial hand posture.

5.1.4. Comparison with other work

To further validate the recognition performance, we tested the proposed system on another public gesture data set – the 10-gesture data set by Ren et al. [30]. According to Cheng et al. [4], Ren et al. were the first to collect 3D static freehand gestures with a depth camera (Kinect) and their gesture data set was quite challenging because the gestures were collected in an uncontrolled environment (cluttered backgrounds and lighting conditions) similar to the one in our study. Therefore, the 10-gesture data set was chosen as the ground truth for the static gesture comparison experiment. The experimental results are shown in Table 4.

Table 4
Static gesture comparison

Ren et al.’s method [30]

Threshold Decomposition Near-Convex Decomposition Our method

Accuracy 93.2% 93.9% 94.5%

	Ren et al.’s method [30]
Accuracy	93.2%	93.9%	94.5%

Table 5

Dynamic gesture comparison

Method	Isolated dynamic gestures		Continuous dynamic gestures

	1-handed	2-handed	1-handed	2-handed
Kristensson’s [17]	93.6%	95.5%	N/A	N/A
Pedersoli’s [28]	84.2%	N/A	N/A	N/A
Ours	95.3%	96.7%	92.4%	90.5%

Overall, our method slightly outperforms Ren et al.’s method for static gesture recognition on the 10-gesture data set. Additionally, Ren et al.’s method requires the user to wear a black belt on his/her wrist when performing gestures. In contrast, we provide a more natural freehand gesture interaction technique for people to use in a real TV viewing environment. From the experimental results, we found that the most confusion between gestures is between gestures 0 and 9, gestures 1 and 8, and gestures 4 and 5. The lower recognition accuracy may be because we used a DDF method to describe the hand region in a binary image (Fig. 2(d)). Compared with other fingers, the thumb is shorter and smaller. As a result, two gestures with and without a thumb may not be distinct enough in a segmented binary image.

For dynamic gestures, we tested recognition accuracy on the following three popular gesture sets: the $1 gesture set [17,28,38], the 26-graffiti gesture set [6,17], and the two-handed gesture set [28]. The $1 gesture set and the 26-graffiti gesture set were used for one-handed gesture recognition tests. In this experiment, we only compared the recognition accuracy of hand motion trajectories among different methods, regardless of the initial static hand postures in the dynamic gestures. The comparison results are provided in Table 5.

As shown in Table 5, our classifier outperforms the classifiers presented in Kristensson et al. [17] and Pedersoli et al. [28] for dynamic gesture recognition on the three public gesture sets. By adopting a conventional HMM method, Pedersoli et al.’s method achieved an average accuracy of 84.2% for isolated one-handed gestures. However, their classifier cannot identify two-handed gestures. Different from an HMM-based approach, Kristensson et al. used a probabilistic reasoning algorithm to incrementally predict the user’s meaningful gestures while still being articulated. Their system achieved a slightly higher accuracy for two-handed gestures (95.5%) than for one-handed gestures (93.6%) due to the additional information in the probabilistic model. By adopting a CHMM method, our classifier achieved an average accuracy of 96.7% for two-handed gestures. The results show that CHMM is quite suitable for two-handed gesture recognition because it can accurately model the interrelationships between two hands.

Additionally, Kristensson et al.’s and Pedersoli et al.’s methods primarily evaluated isolated gesture recognition. That is, their gesture recognition method assumes that the start and end points of a gesture have already been unambiguously spotted in advance in an input video. In contrast, by using the proposed filter model, our system can reject meaningless gestures effectively at the early stage of gesture recognition and spot meaningful gestures accurately, which in turn achieves an average recognition accuracy of 92.4% and 90.5% for continuous one- and two-handed dynamic gestures without prior purification in a natural TV viewing environment.

5.2. Experiment 2: Usability of the user-defined gestures in practice

To investigate the potential of intuitive freehand gestures as an abstract input device for smart TV systems, we conducted Experiment 2 to compare freehand-gesture-based inputs with a conventional remote-control approach. We are interested in the following issues:

Can gesture-based input work effectively?

How do people view gesture-based inputs?

5.2.1. Experimental design

The experiment included three treatments. In Treatment A (TA), participants used a conventional remote control. In Treatment B (TB), participants were asked to use the user-elicited gestures for TV control proposed by Zaiţi et al. [45]. Here, we made a comparison with Zaiţi et al.’s study because a) their research and ours share the same target functions as shown in Table 6, and b) they adopted a similar but slightly different elicitation study method to derive the most popular gesture for each of the target functions. In Treatment C (TC), participants were asked to use the gesture set proposed in our study.

Table 6
Experimental tasks

No. Task name Remote control Zaiţi et al.’s study Our study

1 Turn on the TV Press the “Power” key Open palm Both hands moving from the center middle to the outer left and right

2 Turn off the TV Press the “Power” key Close palm Both hands moving from the outer left and right to the center middle

3 Confirm Press the “OK” key Thumbs-up An “OK” sign

4 Call the main menu Press the “Menu” key Draw letter “M” Spreading hand

5 Mute Press the “Mute” key Close fingers into pinch Hands forming a “T” sign

6 Volume up Press the “Volume +” key Move hand up Move hand up

7 Volume down Press the “Volume −” key Move hand down Move hand down

8 Next channel Press the “Channel +” key Move hand from left to right Move hand from left to right

9 Previous channel Press the “Channel −” key Move hand from right to left Move hand from right to left

10 Go to channel 27 Press digit keys “2” and “7” in succession Draw number “27” in air Draw number “27” in air

11 Search for a specific movie named “2012” Press digit keys “2”, “0”, “1”, and “2” in succession Draw number “2012” in air Draw number “2012” in air

No.	Task name	Remote control	Zaiţi et al.’s study	Our study
1	Turn on the TV	Press the “Power” key	Open palm	Both hands moving from the center middle to the outer left and right
2	Turn off the TV	Press the “Power” key	Close palm	Both hands moving from the outer left and right to the center middle
3	Confirm	Press the “OK” key	Thumbs-up	An “OK” sign
4	Call the main menu	Press the “Menu” key	Draw letter “M”	Spreading hand
5	Mute	Press the “Mute” key	Close fingers into pinch	Hands forming a “T” sign
6	Volume up	Press the “Volume +” key	Move hand up	Move hand up
7	Volume down	Press the “Volume −” key	Move hand down	Move hand down
8	Next channel	Press the “Channel +” key	Move hand from left to right	Move hand from left to right
9	Previous channel	Press the “Channel −” key	Move hand from right to left	Move hand from right to left
10	Go to channel 27	Press digit keys “2” and “7” in succession	Draw number “27” in air	Draw number “27” in air
11	Search for a specific movie named “2012”	Press digit keys “2”, “0”, “1”, and “2” in succession	Draw number “2012” in air	Draw number “2012” in air

5.2.2. Participants and apparatus

We recruited 24 participants from a university campus, including 12 males and 12 females. These participants were between 19 and 51 years old ( $M = 27$ , $SD = 9.3$ ) and came from different professional backgrounds. None of these participants had been involved in the previous study or had any prior experience using vision-based freehand gesture interaction.

We used the same apparatus that was used in Experiment 1.

5.2.3. Procedure

During the experiment, all participants were asked to complete a set of TV tasks including menu navigation and content searching (Table 6) as quickly as possible three times.

Table 6 lists the 11 tasks used in this study. The two left columns of the table identify each task with a sequential number and a task name. The middle column indicates the interactive manipulations by using a remote control. The two right columns indicate the user-defined gestures proposed by Zaiţi et al.’s study and ours for each task. As shown, the user-defined gestures for Tasks 6 through 12 in both studies are the same.

Our experiment was a within-subject design. A Latin square was used to counterbalance different treatment orders. Participants were randomly assigned to these orders.

Before the study, participants were allowed to practice until they successfully completed a task similar to the one in our study. We collected data on user performance and user satisfaction with provided interaction tools. The performance was measured by task completion time, i.e., the time interval between the moment a task started and the moment a participant correctly finished the task. After completing all tasks, each participant was asked to complete a questionnaire concerning their opinions of the three interaction techniques. Participants used a Likert scale (1 worst, 7 best) to evaluate the three techniques in terms of ease of use (i.e., the degree to which a technique can be used to perform a task with effectiveness), ease of learning (i.e., how easy it is to learn how to use a technique), efficiency (i.e., the degree to which a technique can perform a task correctly), intuitiveness (i.e., the degree to which participants can use a technique directly by intuition without rational thought), physical fatigue (i.e., how difficult it is to use a technique physically), and enjoyableness (i.e., the degree to which participants like to use a technique to perform a task).

5.2.4. Results

Figure 7 compares the task completion time for the three treatments. The overall time for TB and TC to complete the given tasks is less than that for TA. As shown in Fig. 7, the average task completion times in TA, TB, and TC are 76.46 s ( $SD = 9.736 s$ ), 32.88 s ( $SD = 5.286 s$ ), and 35.96 s ( $SD = 4.506 s$ ), respectively. Using a one-way ANOVA test, we found that the difference in task completion time between the three treatments is significant ( $F_{2, 69} = 297.791$ , $p = 0.000$ ). Post hoc analysis (Tukey’s HSD) indicated a significant difference between TA and TB ( $t_{23} = 43.583$ , $p = 0.000$ ), as well as between TA and TC ( $t_{23} = 40.500$ , $p = 0.000$ ). No significant difference was found between TB and TC ( $t_{23} = - 3.083$ , $p = 0.276$ ).

Fig. 7.

Comparison of task completion times.

Figure 8 shows the qualitative evaluation results. As shown, both of the freehand-gesture-based interaction techniques (TB and TC) were perceived better than the Remote Control (TA) in terms of ease of use, intuitiveness, and enjoyableness. In comparison, TA was perceived as easier to learn and more comfortable to use than TB and TC. No significant difference was found between TA and TC in terms of efficiency ( $Z_{TA vs . TC} = 0.5$ , $p = .379$ ).

Fig. 8.

Ranking of the three techniques in terms of ease of use, ease of learning, efficiency, intuitiveness, physical fatigue, and enjoyableness.

5.2.5. Discussion

The experimental results show that both TB and TC allow users to interact with the TV easily and intuitively, and both perform significantly better than TA. Our results indicate that TB and TC can also offer improved user experience and user satisfaction from several perspectives.

To more closely evaluate the difference between the three techniques, we found that the required time for participants to perform the task of content searching, e.g., searching for a specific movie, by using TA is significantly longer than that by using TB and TC. This may be because with a remote control, the user has to press arrow keys “←”,“→”,“↑”,and “↓” to select digits or characters from the left panel of the search box (Fig. 9). Thus, the user often must shift their attention between the television screen and the remote control. In comparison, with dynamic gestures to draw Arabic numbers in the air, participants could always keep their eyes on the TV screen and avoid the eye-hand separation problem in TA. Therefore, those gestures are more direct and efficient in performing the given tasks.

Fig. 9.

The user is searching for a movie named “2012”.

The results of subjective rating in our research show that TA is perceived as the easiest technology to learn because it does not require the user to remember gesture semantics defined in TB and TC.

In contrast, TB and TC are perceived as more enjoyable than TA. This may be because TB and TC were designed based on the results of the participatory study of people’s intuitive gestures when interacting with a TV. The user-defined gestures supported by TB and TC involve no complex or strange movements. The simplicity and ease of the gestures, as well as their natural interaction, may contribute to the perceived benefits.

In general, participants showed more positive attitudes toward user-defined gestures proposed by our method than by Zaiţi et al.’s study. For some tasks, such as Turn on the TV, Turn off the TV, and Mute, participants preferred the two-handed gestures designed in our study over the one-handed gestures proposed by Zaiţi et al. One participant stated that

I would like to use the two-handed gesture, both hands moving from the center middle to the outer left and right for Task 1 – Turn on the TV, because it looks like opening a door to a TV show and makes me feel more involved and immersive.

Some participants said that the two-handed gestures could also be used to perform more complex tasks such as Image Scaling and Image Rotating. Previous research has provided empirical evidence on the benefits of practices involving tow-handed gestures in user interfaces [15]. All these findings indicate that two-handed gesture input is a promising technique to improve the directness and degree of manipulation afforded by the user interface.

In addition, participants also noted that the two gestures, Close palm and Close fingers into pinch proposed by Zaiţi et al. for Task 2 (Turn off the TV) and Task 5 (Mute) often caused confusion because the two gestures are very similar and difficult to distinguish. Sometimes, they just wanted to use the gesture to mute the volume, but they would accidentally turn off the TV.

Although TB and TC achieve good scores from several perspectives, some important problems still exist, such as comfort in interaction. It is not surprising to see that participants perceived TA as a more comfortable means than TB and TC. Similar findings were reported by Cabral et al. [3]. A mouse or a remote control can be easily grasped in the palm, with all tasks performed using an index finger or a thumb. In contrast, more muscle groups and longer movements are required in gesture-based interaction. The lengthy interaction may lead to body fatigue. Although in general our participants rated gesture-based input very high, showing the same enthusiasm for new technologies that has been seen in previous research [3,7,15,33], we believe that these problems should be a concern and more efforts are needed to design more comfortable gesture-based commands.

6. Conclusions and future work

Controls for gesture-based interactive TV systems must be accurate, intuitive, physically appropriate and user-friendly. Although considerable work has been reported in recent years for gesture-based TV applications, many of them fall into the category of “a solution looking for a problem”. In this paper, we created a living room environment and adopted a top-down paradigm to explore these qualities in a study of using natural freehand gestures to interact with a smart TV system.

The main contribution of this study lies in the unified framework we proposed for gesture-based applications. This framework has two advantages: 1) We derived a set of 19 user-defined gestures for TV control tasks by using an improved elicitation study method. Compared with many gestural systems designed by experts, this study provides valuable insights into the real needs and expectations of TV viewers and helps designers to learn how end-users would complete a task or utilize a gesture-based system and what types of gestures should be developed in a gesture-based TV system. As a result, the user-defined gesture vocabulary can reduce the risk that a system delivered to end users may be unusable and ineffective. 2) We proposed a unified framework toward the recognition of the user-defined gesture set for smart TV systems. Compared with many existing algorithms developed in a lab environment, this framework focuses on addressing some specific problems that exist in a complex real-world TV viewing environment, including the automatic exclusion of many meaningless actions from TV viewers in their daily life, a smooth switching mechanism between static and dynamic gesture recognition as well as between one- and two-handed gesture identification, and the continuous recognition of multiple dynamic gestures in the air (e.g., a channel switching gesture for channel 127).

To validate the proposed method, we developed a gesture-based TV control system and conducted two experiments and a comparative analysis. Experimental results show that the proposed method can achieve satisfactory recognition accuracy for both static and dynamic gestures, as well as for one- and two-handed gestures. Even in a real TV viewing environment, the gesture-based interaction design allows users to interact with a smart TV system effectively. Participants were also enthusiastic about this new technique.

A limitation of this study is that we did not consider people with limited mobility or a physical disability. Most participants in our study were students and professors from a university aged between 18 and 51. We chose these participants for two reasons: our limited access to subject pools beyond this age group within a university, and our expectation for subjects who can accept and learn new HCI techniques such as freehand-gesture-based interaction with little difficulty. In practice, ordinary users can obtain similar recognition accuracy as shown in Tables 2 and 3 by following the training process illustrated in Section 5.1.

An interesting next step is to study the influence of cultural factors on the design and choices of gestures. Even within the same application domain, different researchers with different cultural backgrounds may obtain different results by running independent elicitation studies. For example, for the same Turn off the TV command, the elicited gestures in Kühnel et al. [18], Vatavu et al. [33], and our study are significantly different. However, for four other commands – Volume Up, Volume Down, Previous Channel and Next Channel – the elicited gestures in these three studies are strikingly consistent. Such cultural differences may lead to more challenges in technical design (e.g., the recognition algorithms). Another interesting future step is to explore a reasonable integration mechanism for freehand-gesture and voice-based multimodal interaction for those with disabilities in TV viewing environments.

Footnotes

Acknowledgements

The authors would like to thank the anonymous reviewers for their insightful comments. This work was supported by the National Natural Science Foundation of China under Grant No. 61772564, 61772468, 61202344 and the funding offered by the China Scholarship Council (CSC).

References

Aly and

Tapus, Prosody-based adaptive metaphoric head and arm gestures synthesis in human–robot interaction, in: International Conference on Advanced Robotics, 2014, pp. 1–8.

Brand,

Oliver and

Pentland, Coupled hidden Markov models for complex action recognition, in: IEEE Int Conf on Computer Vision and Pattern Recognition, Tokyo, 1997, pp. 994–999.

M.C.

Cabral,

C.H.

Morimoto and

M.K.

Zuffo, On the usability of gesture interfaces in virtual reality environments, in: Proceedings of the 2005 Latin America Conference on Human–Computer Interaction, 2005, pp. 100–108.

Cheng,

Yang and

Z.C.

Liu, Survey on 3D hand gesture recognition, IEEE Transactions on Circuits and Systems for Video Technology, 26(9) (2016), 1659–1673.

Comaniciu and

Meer, Mean shift: A robust approach toward feature space analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence 24(5) (2002), 603–619. doi:10.1109/34.1000236.

Elmezain,

Al-Hamadi,

Appenrodt and

Michaelis, Real-time capable system for hand gesture recognition using hidden Markov models in stereo color image sequences, British Journal of Dermatology 144(2) (2008), 424.

Z.Q.

Feng,

M.M.

Zhang,

Z.G.

Pan,

Yang,

Xu,

H.K.

Tang and

Li, 3D-freehand-pose initialization based on operator’s cognitive behavioral models, The Visual Computer 26(6–8) (2010), 607–617.

W.T.

Freeman and

Weissman, Television control by hand gestures, in: Proceedings of IEEE International Workshop on Automatic Face and Gesture Recognition, 1995.

G.W.

Furnas,

T.K.

Landauer,

L.M.

Gomez and

S.T.

Dumais, The vocabulary problem in human–system communication, Communications of the ACM 30(11) (1987), 964–971. doi:10.1145/32206.32212.

10.

Geiger and

Heckerman, A characterization of the Dirichlet distribution through global and local independence, Annual of Statistics 25(3) (1997), 1344–1369.

11.

J.G.

Han,

Shao,

Xu and

Shotton, Enhanced computer vision with Microsoft Kinect sensor: A review, IEEE Transactions on Cybernetics 43(5) (2013), 1318–1334.

12.

Hilliges,

Izadi,

A.D.

Wilson,

Hodges,

A.G.

Mendoza and

Butz, Interactions in the air: Adding further depth to interactive tabletops, in: Proceedings of the ACM Symposium of User Interface Software and Technology (UIST’ 09), 2009, pp. 139–148.

13.

Höysniemi,

Hämäläinen,

Turkki and

Rouvi, Children’s intuitive gestures in vision-based action games, Communications of the ACM 48(1) (2005), 45–52.

14.

Jiwon,

Andrey and

Bertrand, Smart walker: An intelligent robotic walker, Journal of Ambient Intelligence and Smart Environment 8(4) (2016), 383–398.

15.

Kölsch,

A.C.

Beall and

Turk, The postural comfort zone for reaching gestures, Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 47(4) (2003), 787–791.

16.

Kölsch,

Turk and

Höllerer, Vision-based interfaces for mobility, in: Proceedings of IEEE International Conference on Mobile and Ubiquitous Systems (Mobiquitous’04), 2004, pp. 86–94.

17.

P.O.

Kristensson,

T.F.W.

Nicholson and

Quigley, Continuous recognition of one-handed and two-handed gestures using 3D full-body motion tracking sensors, in: Proceedings of the 17th International Conference on Intelligent User Interfaces (IUI’12), 2012, pp. 89–92.

18.

Kühnel,

Westermann,

Hemmert,

Kratz,

Müller and

Möller, I’m home: Defining and evaluating a gesture set for smart-home control, International Journal of Human–Computer Studies 69 (2011), 693–704.

19.

Kumar,

Gauba,

P.P.

Roy and

D.P.

Dogra, Coupled HMM-based multi-sensor data fusion for sign language recognition, Pattern Recognition Letters, 86 (2017), 1–8.

20.

Lee and

Kim, An HMM-based threshold model approach for gesture recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence 21(10) (1999), 961–973. doi:10.1109/34.799904.

21.

W.P.

Lee,

Kaoli and

J.Y.

Huang, A smart TV system with body-gesture control, tag-based rating and context-aware recommendation, Knowledge-Based Systems 56 (2014), 167–178.

22.

Linde,

Buzo and

R.M.

Gray, An algorithm for vector quantizer design, IEEE Transactions on Communications 28(1) (1980), 84–95. doi:10.1109/TCOM.1980.1094577.

23.

Löcken,

Hesselmann,

Pielot,

Henze and

Boll, User-centered process for the definition of freehand gestures applied to controlling music playback, Multimedia Systems 18(1) (2011), 15–31. doi:10.1007/s00530-011-0240-2.

24.

Z.Y.

Mo,

J.P.

Lewis and

Neumann, SmartCanvas: A gesture-driven intelligent drawing desk, in: Proc. IUI 2005, ACM Press, 2005, pp. 239–243.

25.

M.R.

Morris,

Danielescu,

Drucker,

Fisher,

Lee,

M.C.

Schraefel and

J.O.

Wobbrock, Reducing legacy bias in gesture elicitation studies, Interactions 21(3) (2014), 40–45.

26.

M.R.

Morris,

J.O.

Wobbrock and

A.D.

Wilson, Understanding users’ preferences for surface gestures, in: GI’10, 2010, pp. 261–268.

27.

Nielsen,

Störring,

Moeslund and

E.A.

Granum, Procedure for developing intuitive and ergonomic gesture interfaces for HCI, in: GW, 2003, pp. 409–420.

28.

Pedersoli,

Benini,

Adami and

Leonardi, XKin: An open sourceframework for hand pose and gesture recognition using Kinect, The Visual Computer 30(10) (2014), 1107–1122. doi:10.1007/s00371-014-0921-x.

29.

S.S.

Rautaray and

Agrawal, Vision-based hand gesture recognition for human–computer interaction: A survey, Artificial Intelligent Review 43 (2015), 1–54. doi:10.1007/s10462-012-9356-9.

30.

Ren,

J.S.

Yuan and

Z.Y.

Zhang, Robust hand gesture recognition based on finger-earth movers distance with a commodity depth camera, in: ACM International Conference on Multimedia, 2013, pp. 1093–1096.

31.

Ruiz,

Li and

Lank, User-defined motion gestures for mobile interaction, in: Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI’11), 2011, pp. 197–206.

32.

Takahashi,

Fujii,

Naemura and

Satoh, Human gesture recognition system for TV viewing using time-of-flight camera, Multimedia Tools and Applications 63 (2013), 761–783.

33.

R.D.

Vatavu, User-defined gestures for free-hand TV control, in: Proceedings of the 10th European Conference on Interactive TV and Video (EuroITV’12), 2012, pp. 45–48. doi:10.1145/2325616.2325626.

34.

R.D.

Vatavu, A comparative study of user-defined handheld vs. freehand gestures for home entertainment environments, Journal of Ambient Intelligence and Smart Environments 5 (2013), 187–211.

35.

A.J.

Viterbi, Error bounds for convolution codes and an asymptotically optimum decoding algorithm, IEEE Transactions on Information Theory 13 (1967), 260–269. doi:10.1109/TIT.1967.1054010.

36.

Wang,

A.M.

Bernardos,

J.A.

Besada,

Metola and

J.R.

Casar, A gesture-based method for natural interaction in smart spaces, Journal of ambient and smart environments 7 (2015), 535–562.

37.

J.O.

Wobbrock,

M.R.

Morris and

A.D.

Wilson, User-defined gestures for surface computing, in: Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI’ 09), 2009, pp. 1083–1092. doi:10.1145/1518701.1518866.

38.

J.O.

Wobbrock,

A.D.

Wilson and

Li, Gestures without libraries, toolkits or training: A $1 recognizer for user interface prototypes, in: Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (UIST’07), 2007, pp. 159–168. doi:10.1145/1294211.1294238.

39.

H.Y.

Wu,

J.M.

Wang and

X.L.

Zhang, User-centered gesture development in TV viewing environment, Multimedia Tools and Applications 75(2) (2015), 733–760.

40.

H.Y.

Wu,

J.M.

Wang and

X.L.

Zhang, A visual attention-based method to address the Midas touch problem existing in gesture-based interaction, Visual Computer 32(1) (2016), 123–136. doi:10.1007/s00371-014-1060-0.

41.

H.Y.

Wu,

J.M.

Wang and

X.L.

Zhang, Combining hidden Markov model and fuzzy neural network for continuous recognition of complex dynamic gestures, Visual Computer 33(10) (2017), 1227–1263. doi:10.1007/s00371-015-1147-2.

42.

H.D.

Yang,

A.Y.

Park and

S.W.

Lee, Gesture spotting and recognition for human–robot interaction, IEEE Transactions on Robotics 23(2) (2007), 256–270. doi:10.1109/TRO.2006.889491.

43.

H.D.

Yang,

Sclaroff and

S.W.

Lee, Sign language spotting with a threshold model based on conditional random fields, IEEE Transactions on Pattern Analysis and Machine Intelligence 31(7) (2009), 1264–1277. doi:10.1109/TPAMI.2008.172.

44.

H.S.

Yoon,

Soh,

Y.J.

Bae and

H.S.

Yang, Hand gesture recognition using combined features of location, angle and velocity, Pattern Recognition 34 (2001), 1491–1501. doi:10.1016/S0031-3203(00)00096-0.

45.

I.A.

Zaiţi,

S.G.

Pentiuc and

R.D.

Vatavu, On free-hand TV control: Experimental results on user-elicited gestures with leap motion, Personal and Ubiquitous Computing 19 (2015), 821–838. doi:10.1007/s00779-015-0863-y.

46.

H.J.

Zhang,

Kankanhalli and

Smoliar, Automatic partitioning of full-motion video, Multimedia System 1(1) (1993), 10–28. doi:10.1007/BF01210504.

47.

J.Y.

Zhu,

X.Y.

Wang,

W.X.

Wang and

G.Z.

Dai, Hand gesture recognition based on structure analysis, Chinese Journal of Computers 29(12) (2006), 2130–2137.

Function	Gesture	Initial hand posture	Following hand motion trajectory

	Ren et al.’s method [30]

	Threshold Decomposition	Near-Convex Decomposition	Our method
Accuracy	93.2%	93.9%	94.5%

Beyond remote control: Exploring natural gesture inputs for smart TV systems

Abstract

Keywords

1. Introduction

2. Related work

2.1. Freehand-gesture-based systems

2.2. Participatory design of freehand gestures

3. Requirement analysis and user-defined gestures

Table 1 User-defined gesture set for a smart TV system Function Gesture Initial hand posture Following hand motion trajectory

4.2.1. Static feature extraction

4.3.1. Hand posture recognition

5.1. Experiment 1: Performance evaluation of our gesture recognition system

5.1.1. Participants and apparatus

5.1.2. Procedure

5.1.3. Results

Table 2 The confusion matrix of static gesture recognition

Table 4 Static gesture comparison Ren et al.’s method [30] Threshold Decomposition Near-Convex Decomposition Our method Accuracy 93.2% 93.9% 94.5%

5.2.1. Experimental design

5.2.3. Procedure

5.2.4. Results

Footnotes

Acknowledgements

References

Table 1
User-defined gesture set for a smart TV system

Function Gesture Initial hand posture Following hand motion trajectory

Table 2
The confusion matrix of static gesture recognition

Table 4
Static gesture comparison

Ren et al.’s method [30]

Threshold Decomposition Near-Convex Decomposition Our method

Accuracy 93.2% 93.9% 94.5%