Playful Interfaces for Speech Therapy

Brief

The team is investigating ways in which playful interfaces (e.g., video games) can be used to make speech therapy more engaging and therefore more effective.

Speech therapy sessions are laborious and maintaining the motivation of patients is hard, especially when patients are required to perform repetitive monotonous vocalizations themselves at home. Current speech therapy treatments are also not easily accessible to the general public due to cost and demand. We propose combining popular classic game mechanics with speech recognition technology to enable individualised, fun and accessible speech therapy.

This project started with the development of sPeAK-MAN, a Pac-Man-like game with a core gameplay mechanic that incorporates vocalisation of words generated from a pool commonly used in clinical speech therapy sessions. Other than improving engagement, sPeAK-MAN aims to provide real-time feedback on the vocalisation performance of patients. It also serves as an initial prototype to demonstrate the possibilities of using familiar popular gameplay (instead of building an untested game mechanic from scratch) for rehabilitation purposes.

We are currently investigating the recognition of formants to control popular classics like Space Invaders and Tetris. We can reliably disambiguate vowels using this pure quantitative method. Speech invaders and Yak-man (a.k.a. sPeAK-MAN v2) are our current attempts at realising this goal and can be downloaded from the iOS app store:

http://itunes.com/apps/speechinvaders
http://itunes.com/apps/yakman

We are also building a speech game framework that can be used to easily create speech-aware games. The client app will allow patients to choose from a range of games having varying game mechanics to suit their level of competency and preference for genres. The therapist admin app will allow speech pathologists to control the vocalization tasks within speech games.

Here’s the latest game we released built with our game framework:

https://itunes.com/apps/speechwordpop

Members

Dr Chek Tien Tan (Games Studio)
Associate Professor Kirrie Ballard (Speech Pathology, University of Sydney)
Dr Sam Ferguson (Creativity & Cognition Studio)
Dr Andrew Johnston (Creativity & Cognition Studio)
Songjia Shen (Games Studio)
Hemanta Sapkota (Games Studio)

Outputs

C. T. Tan, A. Johnston, A. Bluff, and S. Ferguson, “Speech Invaders & Yak-man : Retrogames for Speech Therapy,” in Proceedings of SIGGRAPH Asia 2014 MGIA, 2014.

C. T. Tan, A. Johnston, A. Bluff, and S. Ferguson, “Retrogaming as Visual Feedback for Speech Therapy,” in Proceedings of SIGGRAPH Asia 2014 MGIA, 2014.

C.T. Tan, A. Johnston, K. Ballard, S. Ferguson, and D. Perera-Schulz, “sPeAK-MAN: Towards Popular Gameplay for Speech Therapy”, in Proceedings of the Australasian Conference on Interactive Entertainment, 2013.

Appeared on UTS Newsroom, 2012.

S. Ferguson, A. Johnston, K. Ballard, C. T. Tan, and D. Perera-Schulz, “Visual Feedback of Acoustic Data for Speech Therapy : Model and Design Parameters,” in Proceedings of the Audio Mostly Conference on Interaction with Sound, 2012.

Oct. 26, 2016, 3:14 p.m.