It is estimated that 120,000 people in Canada and two million in the USA are living with aphasia—a communication disorder most often caused by a stroke that limit their ability to being understood and to participate independently in society. Our Augmentative and Alternative Communication (AAC) mobile application Click AAC aims to help people with aphasia to retell personally relevant events and activities, consequently supporting their social interactions. The application generates a situation-specific vocabulary from photos of events and activities and display it as pictograms that produce speech audio when selected. In such way, the user has the opportunity to find and use appropriate vocabulary to talk about those past events and activities in a timely manner, without needing to navigate an extensive list of words often found in traditional AAC devices
Vocabulary Generation Algorithm
Research on the application of NLP in symbol-based Augmentative and Alternative Communication (AAC) tools for improving social interaction support is scarce. We contribute a novel method for generating context-related vocabulary from photographs of personally relevant events aimed at supporting people with language impairments in retelling their past experiences. Performance was calculated with information retrieval concepts on the relevance of vocabulary generated for communicating a corpus of 9730 narrative phrases about events depicted in 1946 photographs. In comparison to a baseline generation composed of frequent English words, our method generated vocabulary with a 4.6 gain in mean average precision, regardless of the level of contextual information in the input photographs, and 6.9 for photographs in which contextual information was extracted correctly. We conclude by discussing how our findings provide insights for system optimization and usage.
Aphasia and current AAC devices
Aphasia is caused by a brain injury, usually a stroke. Intellectual abilities are relatively preserved, while the severity and pattern of speech and language difficulties varies depending on the brain regions affected. Individuals with Wernicke’s aphasia are able to produce connected speech with complex syntactical forms but lose the ability to comprehend speech, while individuals with Broca’s aphasia present good receptive communication skills, but have difficulties retrieving name of objects, combining linguistic elements, and have a very limited or absent grammar. In the most severe cases, speech is restricted to stereotypical utterances, where the person repeatedly utters a word.
Picture dictionary-style augmentative and alternative communication (AAC) devices can enhance communication and support the social interactions of people with complex communication needs, such as autism, ALS, and aphasia by providing a visual representation of words and phrases and by speaking desired sentences through a synthesized voice (popular AAC devices and mobile applications include Tobii Dynavox, Proloquo2Go, LAMP, TouchChat, and Talk Rocket Go. ). However, communication with an AAC device is extremely slow, creating an enormous disparity that stunts and delays communication, hinders interpersonal interactions, and ultimately leads to the device abandon.
The low communication rate of these AAC devices is in large part due to difficulty organizing the extensive vocabulary needed for the generation of spontaneous utterances in a manner that allows the user to easily find and select desired words when communicating. Since most users of these devices will have difficulty with written language, alphabetical organization is not possible. Instead, vocabulary items are usually organized in a static hierarchy of categories representing superordinate or functional relations (e.g. food → breakfast → croissant; feelings → hungry) that do not reflect common usage (i.e., the subject, verb, and object of any sentence are likely in completely different categories). Even though such organizations are better than most they are nonetheless difficult for people with aphasia as they often have difficulty with lexical and semantic categorizations.
Modern mobile devices offer new potential for designing AAC applications that better attend to the communication needs of people with aphasia. One promising avenue is to use contextual information (e.g. the user’s location, conversation partner) to automatically adapt the vocabulary, reorganizing it to make likely relevant words more easily available. Unfortunately, most efforts to date (both in research and commercial solutions) have required phrases and words to be manually pre-assigned to each context.
Choosing relevant vocabulary and specifying it in the system imposes extra effort on the user and/or caregiver. This task is out of reach for many and quickly becomes impractical for anyone who desires a large vocabulary. As a result, this approach tends to be used for only small sets of contexts and basic communication needs (e.g. ordering food in a restaurant). It does not scale to unexpected situations or unplanned locations, and does not support more nuanced forms of communication, such as telling a story about a past event or activity.