Skip to content

D4: Natural Communication Integration

Project Leaders: Holyfield, Vucetic, Raley, Jakobs, Constituent Board

The project goals are to: (1) Develop technological approaches for integrating dysarthric speech and multimodal natural communication into AAC technology to support individuals with developmental disabilities with and without functional literacy skills and their communication partners; (2) Evaluate the technological approaches developed in collaboration with individuals with developmental disabilities who use AAC and other constituents (e.g., family members, professionals); and (3) Develop freely available technological and clinical guidelines for the approach of integrating natural communication into AAC technology

Man outside in his wheelchair with his AAC system in front of him. The text "Natural Communication Integration" is overlayed.

Problems Addressed: Many AAC users have low-effort natural communication including speech, gestures, and facial expressions. Yet, existing AAC technology is mostly separate from natural communication, limited in its ability to meaningfully integrate unaided communication. It is better to integrate all of a person’s methods of communication!

Goals of the project:

  1. Develop and evaluate prototypes that incorporate natural speech and vocalizations as AAC input.
  2. Explore opportunities to incorporate other natural communicative behaviors such as gestures and facial expressions as AAC technology input.
Two adults sitting on a couch communicating. Slide also includes the goals of the project.

Slide describing how the project will reach its goals, with an image showing early prototype that recognizes speech, transcribes it, and outputs the speech using voice cloning.

How will we accomplish these goals?

  1. Evaluate current prototype that recognizes speech, transcribes it, and uses voice cloning to provide corresponding voice output.
  2. Evaluate prototype that uses the same backend AI with an interface accessible to individuals learning literacy.
  3. Explore approaches to integrating gestures and facial expressions into AAC.
  4. Collaborate with AAC users to iterate on prototypes and to explore new approaches.

Updates

Additional Information