This is an old revision of the document!


Sign Language Recognition

Sign languages are the primary means of communication for the hearing-impaired. They are visual languages and make use of multiple modalities such as hand gestures, body movements and facial expressions to convey information. These modalities are expressed in parallel to form a sign. Signs represent concepts: They are counterparts of words in spoken languages. When expressed one after one in a sign language sentence, coarticulation effects are observed, making segmentation a challenging task.

Sign language recognition is a multidisciplinary research area involving pattern recognition, computer vision, natural language processing and linguistics. It is a multifaceted problem not only because of the complexity of the visual analysis of hand gestures but also due to the highly multimodal nature of sign languages. Although sign languages are well-structured languages with a phonology, morphology, syntax and grammar, they are different from spoken languages: The structure of a spoken language makes use of words sequentially, whereas a sign language makes use of several body movements in parallel. The linguistic characteristics of sign language are different from those of spoken languages due to the existence of several components affecting the context, such as the use of facial expressions and head movements in addition to the hand movements.

Computer Vision-based sign lanugage recognition systems allows the users to input signs to a computer from a natural environment. However they also introduce several challenges, such as the detection and segmentation of the hand and finger configuration, or handling occlusion. Despite the numerous studies in the literature, problems like marker-free hand detection, segmentation, and tracking in unrestricted environments, representation of motion and temporal segmentation of hand gestures are still a challenging problems.

Pilab Projects on Sign Language Recognition

Publications related to Sign Language Recognition

PHD Theses

  • Oya Aran, Vision Based Sign Language Recognition: Modeling and Recognizing Isolated Signs With Manual and Non-manual Components, 2008

MSc Theses

  • İsmail Arı, Facial Feature Tracking and Expression Recognition for Sign Language, 2008
  • Pınar Santemiz, Alignment And Multimodal Analysis In Signed Speech, 2009
  • İlker Yıldırım, “Cooperative Sign Language Tutoring: A Multiagent Approach”, MS Thesis, Bogazici University, 2009.
  • Ahmet Alp Kındıroğlu, “Computer Vision Based Multi-Lingual Fingerspelling Recogntion.”, MS Thesis, Bogazici University, 2011.

Journal Articles

  • Marek H., Pavel C., Dikici E., Kındıroglu A. A., et al. (2010). Automatic Fingersign-to-speechtranslation system. Journal on Multimodal User Interfaces. Volume 4, Number 2, 61-79, DOI:10.1007/s12193-011-0059-3, (2011).
  • Oya Aran, Lale Akarun, “A Multi-class Classification Strategy for Fisher Scores: Application to Signer Independent Sign Language Recognition”, Pattern Recognition, Vol. 43, no. 5, pp. 1717-1992, May 2010.
  • Cem Keskin, Lale Akarun, " Input-output HMM based 3D hand gesture recognition and spotting for generic applications”, accepted for publication, pattern Recognition Letters, vol. 30, no. 12, pp. 1086-1095, September 2009.
  • Oya Aran, M.S. Thomas Burger, Alice Caplier, Lale Akarun, “A Belief-Based Sequential Fusion Approach for Fusing Manual and Non-Manual Signs”, Pattern Recognition, vol.42 no.5, pp. 812-822, May 2009.
  • Oya Aran, Ismail Ari, Alexandre Benoit, Pavel Campr, Ana Huerta Carrillo, Franois-Xavier Fanard, Lale Akarun, Alice Caplier, Michele Rombaut, and Bulent Sankur, “Signtutor: An Interactive System for Sign Language Tutoring”. IEEE Multimedia, Volume: 16 Issue: 1 Pages: 81-93, Jan-March 2009.
  • Oya Aran, Ismail Ari, Pavel Campr, Erinc Dikici, Marek Hruz, Siddika Parlak, Lale Akarun & Murat Saraclar, Speech and Sliding Text Aided Sign Retrieval from Hearing Impaired Sign News Videos , Journal on Multimodal User Interfaces, vol. 2, n. 1, Springer, 2008.
  • S. Argyropoulos, K. Moustakas, A. Karpov, O. Aran, D. Tzovaras, T. Tsakiris, G. Varni & B. Kwon, A Multimodal Framework for the Communication of the Disabled, Journal on Multimodal User Interfaces, vol.2, pp.105-116, Springer, 2008.
  • Alice Caplier, Sébastien Stillittano, Oya Aran, Lale Akarun, Gérard Bailly, Denis Beautemps, Nouredine Aboutabit & Thomas Burger, Image and video for hearing impaired people, EURASIP Journal on Image and Video Processing, Special Issue on Image and Video Processing for Disability, 2007.

Book Chapters

  • Oya Aran, Thomas Burger, Lale Akarun & Alice Caplier, Gestural Interfaces for Hearing-Impaired Communication, in Multimodal user interfaces: from signals to interaction, Dimitrios Tzovaras (Ed.) Springer, 2008.

Proceedings in International Conferences

  • Kindiroglu, A. A., Yalcin, H., Aran, O., Hruz, M., Campr, P., Akarun, L., et al. (2010). Multi-Lingual Fingerspelling Recognition For Handicapped Kiosk. Pria 2010 Pattern Recognition and Image Analysis (pp. 33-37). St. Petersburg.
  • Pinar Santemiz, Oya Aran, Murat Saraclar and Lale Akarun , Automatic Sign Segmentation from Continuous Signing via Multiple Sequence Alignment, Proc. IEEE Int. Workshop on Human-Computer Interaction, Oct. 4, 2009, Kyoto, Japan.
  • Pinar Santemiz, Oya Aran, Murat Saraclar, Lale Akarun, “Extraction of Isolated Signs from Sign Language Videos via Multiple Sequence Alignment,” in Proceedings of the 13th International Conference on Speech and Computer (SPECOM´09), pp. 117-122, St.Petersburg, Russia, 2009.
  • Pavel Campr, Marek Hruz, Alexey Karpov, Pinar Santemiz, Milos Zelezny, and Oya Aran, Input and output modalities used in a sign-language-enabled information kiosk, in Proceedings of the 13th International Conference on Speech and Computer (SPECOM09), pp. 113-116, St.Petersburg, Russia, 2009.
  • Oya Aran, Lale Akarun, “Multi-class Classification Strategies for Fisher Scores of Gesture and Sign Sequences”, International Conference on Pattern Recognition (ICPR) 2008, Florida.
  • Oya Aran, Ismail Ari, Pavel Campr, Erinc Dikici, Marek Hruz, Deniz Kahramaner, Siddika Parlak, Lale Akarun & Murat Saraclar, Speech and Sliding Text Aided Sign Retrieval from Hearing Impaired Sign News Videos , eNTERFACE’07 The Summer Workshop on Multimodal Interfaces, Istanbul, Turkey, 2007
  • Oya Aran, Thomas Burger, Alice Caplier & Lale Akarun, Sequential Belief Based Fusion of Manual and Non-manual Signs , The 7th International Workshop on Gesture in Human-Computer Interaction and Simulation (GW 2007), 2007
  • Thomas Burger, Alexandra Urankar, Oya Aran, Lale Akarun & Alice Caplier, Cued Speech Hand Shape Recognition , 2nd International Conference on Computer Vision Theory and Applications (VISAPP’07), Spain, 2007
  • Oya Aran & Lale Akarun, Recognizing Two Handed Gestures with Generative, Discriminative and Ensemble Methods via Fisher Kernels , Lecture Notes in Computer Science: Multimedia Content Representation, Classification and Security International Workshop, MRCS 2006, Istanbul, Turkey, September 11-13, 2006, Proceedings, , 4015 , 159-166 2006
  • Oya Aran, Cem Keskin & Lale Akarun, Sign Language Tutoring Tool , European Signal Processing Conference, EUSIPCO’05, Antalya, Turkey, 2005

Proceedings in Local Conferences

  • P. Santemiz, O. Aran, M. Saraclar, L. Akarun, Alignment Based Extraction of Isolated Signs from Sign Language Videos, IEEE 17th Signal Processing and Communications Applications Conference (SIU09), Antalya, Turkey, 2009.
  • H. Dibeklioglu, E. Dikici, P. Santemiz, K. Balci, L. Akarun, Sign Language Motion Tracking and Generating 3D Motion Pieces Using 2D Features, IEEE 15th Signal Processing and Communications Applications Conference (SIU07), pp.1-4, Eskisehir, Turkey, 2007.
  • Oya Aran & Lale Akarun, A Particle Filter Based Algorithm for Robust Tracking of Hands and Face Under Occlusion, IEEE 16th Signal Processing and Communications Applications (SIU 2008), 2008
  • Oya Aran & Lale Akarun, Sign Language Processing and Interactive Tools for Sign Language Education , IEEE 15th Signal Processing and Communications Applications (SIU 2007), 2007
  • Oya Aran, Ismail Ari, Amac Guvensan, Hakan Haberdar, Zeyneb Kurt, Irem Turkmen, Asli Uyar & Lale Akarun, A Database of Non-Manual Signs in Turkish Sign Language , IEEE 15th Signal Processing and Communications Applications (SIU’07), 2007
  • Oya Aran & Lale Akarun, Combination Strategies for 2D Features to Recognize 3D Gestures , IEEE 14th Signal Processing and Communications Applications (SIU 2006), 2006
  • Oya Aran, Cem Keskin & Lale Akarun, Engelliler için Bilgisayar Uygulamaları ve Işaret Dili Eğitimi , GAP V. Mühendislik Kongresi Bildiriler Kitabı, 2006

Software

Databases

  • BUHMAP-DB: A video database of non-manual signs where both facial expressions and global head motion are included.