Billy is a non-verbal 3-year-old with autism spectrum disorder (ASD). When Mom and Dad try to brush his teeth, he bites them. When they try to get him dressed, he kicks and punches. Billy is obviously unhappy, but what is he trying to say?

This scenario is all too familiar for many families learning to communicate with non-verbal members, but there is help: researchers like Ralf Schlosser.

Schlosser, assistant dean of research in the School of Clinical and Rehabilitation Sciences at Northeastern University, has spent his career researching augmentative and alternative communication (AAC) for those with severe speech, language and literacy difficulties. He specializes in helping children on the autism spectrum.

He believes families play an imperative role in developing communications technology.

“Sometimes families may have an advantage over clinicians or teachers in that they are likely very familiar with their child’s idiosyncratic communicative behaviors such as gestures or vocalizations and have the ability to ‘read’ these and respond appropriately.

As a result, it may sometimes feel unnecessary to use an AAC system,” he says.

IMAGE: Shutterstock

But families aren’t always together, and their methods may not work outside the home. These coping methods might, in fact, even hinder the child’s ability to communicate beyond the family circle. This is where technology can help.

Family involvement

Oliver Wendt of Purdue University emphasizes that family and user feedback are critical in AAC development. “You can no longer develop AAC technology without it,” he says.

AAC development involves co-design with direct input from users and stakeholders, usability input from families and therapists and assessments of tech used during interventions. Comments about an app’s usability, like “the interface needs to be simpler” or “it was a lot to navigate,” are crucial. “You need this type of feedback to fine-tune the app and ensure uptake within the community,” Wendt says.

From a design perspective, technology for verbally challenged individuals isn’t a one-size-fits-all solution because each person’s challenges are different.

“Technology opens the doors to interact with other individuals. Use of speech output along with interaction efficiencies is going a long way to make communication possible in schools, at work and in the community,” Schlosser tells KUST Review.

Another vital tool, Schlosser says: the “written gloss.” In a symbol-based AAC app or device, there are graphic symbols like an image representing the concept “drink” or “hungry” next to a corresponding word or phrase. This helps clarify the symbol’s meaning, aids in language learning and helps communication partners understand the user’s message.

He says speech-generating devices and tablets with AAC-specific apps allow individuals to talk using speech, digitized and/or synthetic. Tech also offers efficiency in word retrieval, whether singular or combined into a sentence.

One such tool is GoTalkNow.

GoTalkNow is a comprehensive, iPad-compatible AAC app based on the Project CORE research from the University of North Carolina’s Center for Literacy and Disability Studies. It includes core vocabulary — a collection of most of the words that make up typical daily communication that the user can customize by creating and adding pages. For those with literacy challenges, the app also includes symbols, the option to use images from the users’ cameras and a built-in internet-image search.

“Record video modeling sequences to pair instruction with communication. Play music to heighten interest. Choose text-to-speech or recorded audio for message content. Jump to any communication page, even from a different communication book. Add auditory cues to allow both auditory and visual scanning. Select automatic or step scanning, adjust scanning speed, and group buttons together by row or column for more efficient scanning,” the website advertises.

Skills required

Schlosser says many AAC tools require additional skill sets.

Users must depend on traditional writing and spelling or understand the graphic symbols used to represent concepts. They also need to understand the tool’s functionality and learn to use the program during interactions.

Schlosser says there are obstacles for end users. These include the facts that applications don’t always meet all of an individual’s communication needs and vocabulary selection can further overburden already busy support professionals. “Vocabulary selection in AAC for AAC applications and speech-generating devices is generally a time-consuming process, and school-based speech-language pathologists have little time given their caseloads,” Schlosser says.

But it appears that AI and machine learning can help alleviate some of these issues.

A recent paper by Schlosser, Mauricio de Fontana and Christina Yu, the developers of the QuickPic AAC app that uses AI image analysis to lay out vocabulary, explores two algorithms (ChatGPT-3.5 and NLP-AAC) and found that ChatGPT-3.5 provided more relevant and accurate vocabulary.

The app analyzes an image of an activity and produces relevant words represented by picture communication symbols and organizes them in left to right categories in this order: subject, verb, attribute, object.

“This allows creating the displays on the fly or just in time, a process that was previously unimaginable. Just-in-time supports have various cognitive advantages for the learner and allow instructors to capitalize on teachable moments. Within minutes, the clinician can then customize the display as needed,” Schlosser says.

A future for wearables

Schlosser’s research has recently crossed 10,000 citations (uncommon within his field). It reflects long-term influence across research, and he is excited about what AI will bring to the table in the future. The research has only scratched the surface of its impact on AAC.

He sees many opportunities for wearable technologies and has done some small studies in this area, including tools like smart watches.

Researchers were concerned the watches’ small size might be an issue, but the individuals in the study all responded well, he says.

Another study included remote visual interventions like photos, video or text, in real-time, to assist a learner.

Schlosser also says measuring physiological responses like heart rates could also help manage anxiety and support communication.

Wendt cautions that smart watches are still too expensive for many users, but he is enthusiastic about other wearables. People on the autism spectrum with verbal challenges also frequently experience sensory impairments, so the possibilities are endless as the wearables field moves away from watches and toward tech that could be directly woven into clothing, he says.

Help from AI

And as with most industries around the globe, AI will further individual targeted tech design for AAC.

“Diagnostics and prognosis will be more effective through AI. AI can compare the communication profile of a user with data from a plethora of other users with similar conditions. This will help to determine how severely someone is involved and what the prognosis/outlook is,” Wendt says.

“Future AAC applications may be able to predict what vocabulary is needed by a user and then create that content right on the spot.”

Innovations in AR and VR in tandem with AI will only further tech development in this space, as devices become smaller and more user-friendly, they become more cost-effective. “This will allow users to learn and interact in completely new environments that allow new opportunities to learn and grow,” he tells KUST Review.

KU Autism study

With the increasing diagnoses of autism spectrum disorder around the world comes a desperation to understand its origins. While experts believe causation is multi-faceted, researchers at Khalifa University led by Hamdan Hamdan and collaborators including Fakih-IVF Fertility Center and Baylor College of Medicine are studying the condition’s genetic components.

The team has identified seven novel mutations in autistic children within the UAE population using next-generation sequencing.

“The investigation aims to uncover the roles of these novel genes as well as the complex mechanistic pathways involved in ASD using advanced techniques, such as CRISPR-Cas9 gene editing and BioID,” Hamdan says.

The CRISPR-Cas9 gene editing and BioID protein mapping individually have their challenges, but in combination could help identify how mutations contribute to autism and brain development.

The team hopes that identifying mutations specific to the Emirati population may contribute to personalized, targeted therapies and offer important findings for global ASD research highlighting the importance of population-specific studies in understanding the full spectrum of genetic variations associated with autism.

More like this: Autism checks