Robot and avatar based interactive system for children's education

The aim of this thesis is to present usable educational tools of students with special needs. Hearing impaired students cannot use text and speech based technological, educational material that is becoming a crucial tool for modern education. While they are not a direct substitute for human teachers, technological support materials in education have been shown to motivate children and aid instructors. To allow the use of such methods, three applications aimed at educating deaf children using the robot, the web, and avatar-based technologies was designed. One such technological medium that can be used to connect with such children are robots. Robots are often used in children's education by making use of their inherent personified nature compared to other non-human teaching mediums. Children prefer to interact by touching during a training session. In this study, we developed a method, where two humanoid robots namely R3 and NAO are used to perform sign language gestures to communicate with children using sign languages with creating human-like hand, arm and body gestures. The most powerful aspect of the robot is its embodiment that causes children to interact with the robot socially, rather than treating it as a simple, book like education tool. One other educational tool we have developed is a signing avatar. Like videos of signers, a signing avatar for TİD would allow communicating information in the form of visual gestures, thus becoming usable when the use of text is infeasible. However, compared to videos, animated avatars offer several advantages like easy reproducibility of gesture sequences, control of point of view, adjusting the speed of the sign and being smaller in storage and bandwidth then videos. For this thesis, a signing avatar was developed to represent a portion of the social sciences course for Turkish primary education curriculum. Finally, a web and android based experimental framework was developed to perform customized human-machine interaction experiments. Sign language and drum based interaction game schemes were developed to perform in-depth analysis of different education based modalities with users from different demographics. By performing usability studies and evaluations in a gamification, scheme using turn-taking games, the effectiveness of the described three methods were analyzed. The observations and the reactions of children demonstrate that of the three technologies implemented, robot technologies may be the most effective method for assisting children. While a prototype interaction scheme and used in controlled education settings have been constructed, in their current technology level, price range and ease of operability robots such as Nao do not present an implementable option of presenting the entire primary school corpora. For that reason, a signing avatar and recreated content from the Social Sciences corpora have been implemented. While the signing avatar an imitation of videos created by signing users, it possesses the advantage of easily synthesizing novel sequences of content from existing isolated sign videos. To analyze the success and effectiveness of the developed methods, the performance of the text, signer video and avatar-based interaction in a human-computer interaction scheme using elementary school students have been compared. The results demonstrate that of the three methods, video-based methods were the most comprehensible closely followed by avatar based methods among kids who did not know how to read. Since the aim was to make the signing avatar as comprehensible as possible, the results demonstrate that goal have been.