Chinese Sign Language Recognition System

With the rapid expansion of computer's influence in the modern society, high performance, high intelligence and high usability are generally regarded as the main trend of the current computer science development. Especially, under the rapid developments in computing technology, communication and display technology, the increasing limitations of the traditional human computer interaction technology based on mouse and keyboard become more and more apparent in the updating display technology, virtual reality and wearable computer. The research goal of multi-perception machine technology is to develop the core technology to resolve the problem of high intelligence and high usability of computing device and to create the harmonious and natural human-computer interaction (HCI) environmentThe key issue is to make the computer precisely perceive the different human expressing means including human natural language, gesture language and facial language. Sign language recognition (SLR), as one of the important research areas of human-computer interaction (HCI), has spawned more and more interest in HCI society. The aim of sign language recognition is to provide an efficient and accurate mechanism to transcribe sign language into text or speech so that communication between deaf and hearing society can be more convenient. From a user's point of view, the most natural way to interact with a computer would be through a speech and gesture interface. Thus, the research on sign language and gesture recognition is likely to provide a shift paradigm from point-and-click user interface to a natural language dialogue-and-spoken command-based interface.

Sign language as a kind of gestures is one of the most natural ways of exchanging information for most deaf people. Chinese sign language (CSL) is a language of choice for most deaf people in China. Chinese sign language can be classified into two categories. One is finger spelling, and the other is hand gesture where each gesture corresponds to one Chinese word or phrase. Currently, the CSL dictionary contains about 5500 conventional Chinese gestures including postures and gestures.

The research on SLR has many applications, for example:
1) Sign language recognition make the communication between the hearing disabled and the hearing abled possible
2) From the cognitive point of view, the research on the mechanism of understanding human vision language can improve the computer intelligence to understand human language.
3) Agent of virtual reality can be controlled by hand gesture.
4) Demonstration learning of robot.
5) Multi-modal interface in virtual reality and augmented reality.
In summary, the research on sign language recognition not only has theoretical values, but also wide application areas.

Research Focus/Issue (Top)

SLR can be classified into two classes according to the devices used to capture gestures, i.e. vision based SLR and Dataglove based SLR. Vision-based SLR utilizes cameras to capture the video (frame) images of hand gestures. The advantage of this approach is that the signer does not have to wear any complex powered input devices and the disadvantages are its instability and impreciseness due to poor illuminant conditions and limited computing power in popular computers. Furthermore, the vision-based SLR has a difficult time performing the task of large vocabulary SLR, because many technical issues on image understanding are still open or need to improve. On the contrary, Dataglove based SLR measures hand gestures using direct devices such as Datagloves and position-trackers. The advantage of this approach is that it captures gesture data robustly and extracts features for further recognition in real-time using less computing power and the disadvantage is that the signer has to wear the device.

For the purpose of the widespread use of large vocabulary SLR system, two research works are in progress as follows.

1. Large-vocabulary signer-independent isolated sign words and continuous sign sentences recognition system over a vocabulary of 5100 signs using the Cyber-gloves and position-trackers as data input devices.
a) Effective feature extraction from different signers. (PCA, transformation to frequency domain)
b) The solution to the movement epenthesis problem. (Transition models, segmentation-based)
c) Compact training sentences for a general model.
d) Minimum unit definition in SLR and its extraction.
e) The use of statistical sign language models.
f) The utilization of non-manual parameters in sign language. Non-manual parameters in sign language include gaze, facial expression, mouth movement, position and motion of the trunk and head.

2. Medium vocabulary isolated sign language recognition system using PC camera as input device, where gesture features information is extracted from gesture video (frame) images.

Research Achievement (Top)

The project of Chinese SLR is one of the sub-projects of "muti-modal perceptron machine". This research was supported in part by Natural Science Foundation of China (Grant No. 69789301), National Key-Basic Research Initialize (Grant No. 2001cca03300) and National High-Technology Development '863' Program of China (Grant No. 2001AA114160). The projects have been authenticated and won the second prize ofscience and technology from Beijing. One patent has been applied and now in review.

In signer-dependent SLR, the average recognition rate of 94% is achieved on the recognition of 5100 CSL isolated signs and 90% on the recognition of 1000 continuous sign language sentences.

In signer-independent SLR, the average recognition rate of 91% is achieved on a vocabulary of 5100 signs, where signer data are collected from 6 signers. The best accuracy of 91.3% can be gotten for continuous sign language recognition.

For vision-based sign language recognition, an accuracy of about 90% on the vocabulary size of 450 is achieved with the aid of colored cotton gloves.