ExTOL: End to End Translation of British Sign Language

Project Overview
Publications




ExTOL is an EPSRC funded project (EP/RO3298X/1) which brings together the Centre for Vision Speech and Signal Processing at the University of Surrey, with the Visual Geometry Group at the University of Oxford and the Deafness Cognition and Language Research Centre at UCL.

About the project:


Linguistic study of sign languages is quite new compared to spoken languages, having begun only in the 1960s. Linguists are very interested in sign languages because of what they can reveal about the possibilities of human language that don't rely at all on sound. One of the problems is that studying sign languages involves analysing video footage - and because sign languages lack any standard writing or transcription system, this is extremely labour-intensive. This project will develop computer vision tools to assist with video analysis. This will in turn help linguists increase their knowledge of the language with a long term ambition of creating the world's first machine readable dataset of a sign language, a goal that was achieved for large amounts of text of spoken language in the 1970s.

The ultimate goal of this project is to take the annotated data and understanding from linguistic study and to use this to build a system that is capable of watching a human signing and turning this into written English. This will be a world first and an important landmark for deaf-hearing communication. To achieve this the computer must be able to recognise not only hand motion and shape but the facial expression and body posture of the signer. It must also understanding how these aspects are put together into phrases and how these can be translated into written/spoken language.

Although there have been some recent advances in sign language recognition via data gloves and motion capture systems like Kinect, part of the problem is that most computer scientists in this research area do not have the required in-depth knowledge of sign language. This project is therefore a strategic collaboration between leading experts in British Sign Language linguistics and software engineers who specialise in computer vision and machine learning, with the aim of building the world's first British Sign Language to English Translation system and the first practically functional machine translation system for any sign language.

Other info:
Surrey Press Release
Limping Chicken Article (BSL and English)

Institution Lead Academic
University of Surrey (lead)
Prof Richard Bowden
University of Oxford
Prof Andrew Zisserman
University College London
Dr Kearsy Cormier
Prof. Bencie Woll
Advisory Board Robert Adam (British Deaf Association)
Onno Crasborn (Radboud University Nijmegen)
Sarah Ebling (Interkantonale Hochschule für Heilpadago)
Thomas Hanke (University of Hamburg)
Andrew McParland (BBC)
Mark Wheatley (European Union of the Deaf),