Designing Interactive Graphical Interfaces for Teaching Japanese Manual Alphabet

Miki Namatame1), Yasushi Harada2), Fusako Kusunoki2) and Takao Terano3)

1) Department of Design, Tsukuba College of Technology
2) Tama Art University
3) Tokyo Institute of Technology 4-3-15, Amakubo, Tsukuba, Ibaraki, Japan
miki@a.tsukuba-tech.ac.jp

Abstract: This paper introduces Practice! Yubimoji AIUEO (PYA), which enables not only deaf pupils but hearing pupils to learn basic character expressions (AIUEO) of the Japanese manual alphabet (Yubimoji). Learning Japanese manual alphabet (or manual KANA) also serves to present a "barrier-free" concept to hearing pupils. Typically used on a personal computer, PYA has an easy-to-use visual interface. This paper describes the principles of PYA, how it works, and why PYA is effective from an "edutainment" point of view. The principles of PYA are applicable to the manual alphabet of every language. This Yubimoji-software "Practice! Yubimoji AIUEO (PYA)" was published by Shogakukan Inc. which is a famous publisher in Japan, because of its usability and intelligibility. 700 packs of this software have been donated to all of the domestic schools for the deaf, and the many educational facilities for hard-of-hearing people.
Key Words: Barrier-free Education, Edutainment (education+entertainment), Manual Alphabet, Universal Design, Visual Interface

1. Introduction
 Yubimoji is a communication method used by deaf and hard-of-hearing people in Japan. Yubimoji represents syllables (i.e., the 46 Japanese sounds) in the same way as do the hiragana and the katakana. It is the system borrowed from the written language. Each of 46 kana characters can be represented manually, i.e., as a manual alphabet. So we define Yubimoji as "manual kana". Yubimoji (manual kana) conveys only a specific sound, rather than a meaning. Sign language uses hand shapes to convey meaning, while Yubimoji (manual kana), the manual alphabet, conveys only the letters of the alphabet and is used to assist the sign language.
 Japanese sign language and Japanese manual alphabet (manual kana), are not commonly used by hearing Japanese people. The language and alphabet system is locally created, and is quite different from English. To promote the language system, we are conducting a project to develop a PC-based edutainment system aimed at pupils of elementary school or kindergarten age, those who are just beginning to learn Japanese manual alphabet. Manual alphabet (manual kana/Yubimoji) is not necessarily learned in schools for the deaf and hard-of-hearing in Japan. But, the authors think it is effective in order to teach that a word is made of syllables. It is a special feature of the Japanese language and kana system. Learning Japanese manual alphabet (manual kana) also serves to present a "barrier-free" concept to hearing pupils. It is important to have pupils learn about disabilities and normalization in order to build a barrier-free society.
 Despite the rapid development of computer technologies, computer support for sign languages or manual alphabet has not been a popular topic until recently. Most of the current support systems focus on highly motivated users, and as a result they have many sophisticated functions. For example, studies reported in [1] and [2] emphasize networked communications for using sign language systems. The Mimehand and Mimehand II systems in [3] utilize language representation movements of animated agents, which is very complex to implement. In S-Tel [4], avatar type characters communicate using a representation of the sign language. Virtual RadLab [5] provides school children with a large-scale virtual environment for learning.
 To date, few studies have been conducted from the pedagogical and computerhuman interaction points of view. To create an easy-to-use interface for beginners, we have developed a learning system called "Practice! Yubimoji AIUEO (PYA)". PYA enables pupils to learn basic character expressions (AIUEO) of the Japanese manual alphabet (Yubimoji). The main focus of this paper is to demonstrate the effectiveness of this simple but interesting system in effectively educating these pupils.

2. Principles of PYA
 As described in the previous section, most of today's sign language learningsupport systems are designed for highly motivated people. This means that the digital content is usually manipulated via keyboard and/or menus, using a built-in dictionary that may be searched for specific words or verbs. Some systems use animated images to display the content. Current systems often emphasize the amount of content and various query-processing functions.
 Such functions are appropriate for motivated and advanced users, but beginners find the language system very hard to understand. In contrast to these conventional learning support systems, our system will motivate hearing people, especially elementary school pupils, to learn the Japanese sign language system. In the first stage, we focus on only the Japanese KANA characters. In practical educational environments for deaf and hard-of-hearing people, the integration of KANA characters, finger shapes for the manual alphabet, corresponding mouth forms, and sounds is required.
 Turning the palm in the correct direction is critical, because the shape represents a different sound when the direction is incorrect. Specifically, the forms of the manual alphabet must be understood from the standpoint of both speaker and reader (or listener). For manual alphabet, the speaker and the reader/listener are similar to the obverse and reverse sides of a coin. We must learn not only particular manual shapes, but also the appropriate palm direction for each. We often forget that the forms appear reversed when seen from the opposite direction. We have difficulty interpreting the reversed character, or are confused. In order to solve this problem, existing learning systems enable users to select the speaker's or the reader's/listener's standpoint. However, these conventional learning systems are difficult for hearing pupils and beginners to use. Beginners, lacking experience with sign language, find it hard to understand that manual shapes have two appearances. This realization can only come from an unconscious and intuitive understanding of sign-language communication. This seemingly simple observation presents a serious design issue for the software implementation team.
 PYA is equipped with a simple, web-based interface that will attract elementary school pupils. PYA presents the speaker and reader/listener standpoints simultaneously. (For the reader/listener, the face, voice, and hand shape are portrayed; for the speaker, only the hand shape is shown.)
 Aside from PYA, there are very few educational courses available for hearing students, and these tend to use conventional textbooks. These textbooks are not particularly attractive, and can be quite difficult to understand. The PYA interface is easy to use, partly because the directions of the manual shapes are much clearer than the ones in conventional textbooks.
 Based on the above discussion, the key aspects of PYA may be summarized as follows:
(1) Visual representation of the various characters from both the speaker side and the reader/listener side;
(2) Simultaneous displays of corresponding Kana characters, manual shapes, mouth forms and sounds;
(3) Integration of manual shapes, sounding faces, and the corresponding animation;
(4) Animated graphics showing manual movements; and
(5) Explanations of the origins of the manual shapes for the various characters.

Fig. 1. The standpoint of both speaker (left) and reader (right): for example ”あ(A)”

3. How PYA Works
 Figure 2 shows how PYA implements the five principles discussed in the previous section. The integrated visual interface supports quick learning for beginners.
 The first display in Fig.2 is the main screen of PYA, which shows all of the Kana characters and three kinds of shift icons. Each Kana character acts as a button, allowing the user to display the corresponding manual alphabet.
 The second display appears when a user points the mouse to place 'あ(A)' PYA then automatically shows the third display when the mouse is clicked.
 The girl appearing in the third display pronounces the sound of 'あ(A)' At the same time, she demonstrates the corresponding finger shape with the appropriate movements. The central finger form represents the character as seen by the reader/listener, and the larger finger form represents the character as seen by the speaker. These simultaneously displayed finger forms enable the user to easily grasp the correct form for 'あ(A)'. These displays are designed based on principles of (1) through (4) above.
 If there is information available about the origin of the manual shape, the puppy will wag its tail. The fourth display will then appear to explain how the form was determined. Such help information is essential for the following reasons:
— The manual alphabet are designed in an ad hoc manner, and as a result there are no fundamental principles and
— Beginners tend to become confused about the palm directions for each character shape, making it difficult to memorize the shapes.
 This help information provides an example of principle (5) from the previous section in action.

Fig. 2. How PYA works

4. Experiments
 Using PYA, we conducted three-hour classroom experiments at Sumiyoshi Elementary School in Kobe in December 2003 and 2004. The experiment took the form of a special lecture on "integrated courses". The objective of this special lecture was to give the children a deeper understanding of disabled people. The experiment consisted of an oral lecture and a practice session using PYA. The oral lecture was given by the first author, who is an expert on the education of deaf and hard-ofhearing students, from Tsukuba College of Technology.
4.1 Experimental Setups
 In Japan, experiments such as this normally would not be permitted in an elementary school. In this case, from an experimental perspective, the design was compromised by the imposed requirement that we provide the same content and the same teaching style to each of the students. This requirement prevented us from directly comparing the efficacy of PYA as compared to conventional oral lectures.
 A total of 41 pupils, all 11 or 12 years old, were selected to be the subjects of the experiment. They are all hearing students. Five of them had had educational experiences outside of Japan.
 They were divided into two groups: Group 1, with 21 pupils, and Group 2 with 20 pupils.
 We gave Group 1 pupils PYA practice first, followed by an oral lecture. Group 2 pupils were given an oral lecture first, then PYA practice. We gave each group the same problems, but in a different order.
 The 50-minute oral lecture included an explanation of the importance of the sign language and manual alphabet system as well as an exercise to help remember how to spell each subject's own name and some simple activities based on certain popular words. Figure 3 shows the oral lecture: the introduction of manual alphabet (manual Kana/Yubimoji), and the education of pupils including individual examinations. In the oral lecture, the lecturers emphasized the importance of both mouth forms and finger forms.
 The PYA practice, also 50 minutes long, consisted of the free use of the system along with some simple exercises based on certain popular words. In the practice session, pupils freely explored the visual interface of PYA in an attempt to memorize how to represent simple words and their own names. Figure 4 presents two photographs; (a) is an example of individual exercise and (b) is a photograph of a collaborative learning activity.
 After the practice and the lecture, we conducted individual performance testing to determine how well students understood the language system.
4.2 Methods for Performance Testing
 The individual performance tests ask the student to display manual alphabet for the following.
(1) The student's name; difficulty varies depending on the name.
(2) Simple words, such as 'いえ(IE)' (house), which they were taught in the first stage. These words all use only the first five characters of the Kana table.
(3) 'はる(HA-RU)' (spring), which they were taught earlier.
(4) 'なつ(NA-TU)' (summer), which they were also taught. The forms for this word are somewhat confusing.
(5) 'みるく(MI-RU-KU)' (milk), which they had been taught. The forms for this word are very confusing because the directions are difficult to represent.
(6) 'あしたあおう (A-SI-TA-A-O-U)' (meet tomorrow), which they must compose using their own knowledge.
 It took 2 to 5 minutes for each pupil to complete the tests.
 The results were evaluated for the correctness of the character forms (0-15 points), the direction of the forms (0-5 points), and the smoothness of the movements (0-5 points).
4.3 Results and Discussion
 From our observations during the classroom experiment, we confirmed the following.
(1) When using text materials, the children became confused about the character direction. In contrast, when using PYA, they did not become confused and enjoyed participating in collaborative practices.
(2) From the performance tests, we acquired 40 samples (21 from Group 1 and 19 from Group 2). Various evaluation items (rated 0 to 25 points) are summarized in Table 1.
(3) The total average score was 7.00, with a standard deviation of 4.43. The fact that the scores cover such a wide range suggests that these tests are not easy for the students. The students were categorized under two or three headings depending on group membership, understandability, difficulty, and interest. The categories are as follows.

Fig. 2. How PYA works
Fig. 3. Photograph of the oral lecture by the first author
Fig. 4. Photographs of PYA Practice
Table 1. Summary of the Experiments

 Based on Table 1, we can report the following findings.
1) Recent use of PYA results in better learning performance (Group 1, 6.33 VS Group 2, 7.74). This means that PYA is appropriate as a complement to other learning materials for the pupil.
2) The high understandability for both the lecture and PYA indicates high performance (PYA Understandability High, 7.74; Lecture Understandability High, 7.04). However, higher interest does not affect performance (PYA Interest High, 6.55; Lecture Interest High, 6.87). This means that high understandability for the course is essential for learning.
3) Both the high- and low-difficulty scores for the course indicate lower performance (1st Course Difficulty High 6.50, Low 7.33; 2nd Course Difficulty High 6.14, Low 6.20). This suggests that an optimal difficulty level for the course does exist.
4) The higher interest scores indicate lower performance (PYA Interest High, 6.55; Lecture Interest High, 6.87; 1st Course Interest-High, 6.63; 2nd Course Interest-High, 6.48). This means that over-attractiveness could reduce the learning effects.
 From our experiments and preliminary statistical analyses, we have concluded that PYA is a good tool for learning. However, a more comprehensive analysis is needed to reveal the characteristics of our learning courses.
 From the questionnaire data, we find that most of the subjects have a special interest in manual alphabet (manual KANA/Yubimoji) and have a desire to communicate with deaf and hard-of-hearing people. The experiments were thus considered successful from the pedagogical viewpoint.
 The importance of the concepts of situation [6] and/or the cultural-historical approach [7] for learning new material is often emphasized. Learning does not refer to individual knowledge acquisition but rather to collaborative knowledge-related activities. Therefore, situations involving collaborative activities are most important.
 Although the current implementation of PYA only supports individual learning, we have often observed students enjoying emerging collaborative practices during our experiments. This could lead us towards projects involving total learning environments with mutual communication and other collaborative environments [5], [8], [9], [10].

5. Concluding Remarks
 In this paper, we introduced a PC-based Japanese manual alphabet (manual Kana/Yubimoji) learning system called PYA and have discussed the efficacy of its visual interface. The current demonstration of PYA is available on our web site [11]. This Yubirnoji-softwzre "Practice! Yubimoji AIUEO (PYA)" was published by Shogakukan Inc. which is a famous publisher in Japan, because of its usability and intelligibility. 700 packs of this software have been donated to all the domestic schools for deaf and hard-of-hearing, and the educational facilities for deaf and hard-of-hearing people by "The Foundation for the Advancement of Juvenile Education in Japan" (財団法人　日本児童教育振興財団）

 PYA is a very simple system from both the technical and educational points of view. During the experiments, the pupils clearly enjoyed learning while using the system. We conclude that PYA can be effective as an "edutainment" tool at the elementary level. The principles of PYA are applicable to the manual alphabet of every language.
 The most important areas to be explored in the future are the development of (1) a collaborative framework for learning and (2) new design principles for implementing similar systems. Further research also includes enhancing the functionality of PYA to add the capability of utilizing 3-D visual interfaces. In addition, to further examine the characteristics of student behavior, we will conduct experiments to observe detailed motions of the subjects using, for example, motion capture devices.

6. Acknowledgments
 The authors wish to thank Professor S. Inagaki, Faculty of Human Development, Kobe University and the staff of Sumiyoshi Elementary School for their collaboration with the experiments, and wish to thank Y. Hirano, chief editor of Shogakukan Inc. and "The Foundation for the Advancement of Juvenile Education in Japan" for their hearty supports.

References
1) Magnussen, M.: Videotelephony in Intervention with People with Speech and Language Disabilities. 12th Annual Conference, Technology and Persons with Disabilities, MAGNU2_T.TXT (1997) 8.
2) Sperling, G.: Future Prospects in Language and Communications for the Congenitally. In Deaf Children: Developmental Perspectives, Academic Press (1978) 103-114.
3) Mimehand web-page (2001): http://www.hitachi-eu.com/egov/case_studies/mimehandl.htm (in English), http://www.hitachi.co.jp/Prod/comp/app/shuwa/ (in Japanese).
4) Kuroda, T., Sato, K., and Chihara, K. Development of S-Tel: Avatar-Type Hand Language Transmitting System (in Japanese). Journal of Virtual Reality Society in Japan, Vol. 3, No.2 (1998) 41-46.
5) Virtual RadLab Web-Page (2003): http://www.virat.nottingham.ac.uk
6) Lave, J., and Wenger, E. Situated Learning: Legitimate Peripheral Participation. Cambridge University Press, 1991.
7) Cole, M., and Engestrom, Y A Cultural-Historical Approach to Distributed Cognition. In Salomon, G. (ed.): Distributed Cognitions: Psychological and Educational Considerations. Cambridge University Press, (1993), 1-46.
8) Hammond, N., and Allinson, L. Travels around a Learning Support Environment: Rambling, Orientating or Touring? Proc. CHI 1988, (1988) 269-273.
9) Hudson, J. M., and Bruckman, A. Effects of CMC on Student Participation Patterns in a Foreign Language Learning Environment. Proc. CHI 2001, (2001) 263-264.
10) Sherman, L., et al. StoryKit: Tools for Children to Build Room-sized Interactive Experiences. Proc. CHI 2001, (2001) 197-198.
11) Harada, Y: PYA Web-Page. (2004) http://ss4.idd.tamabi.ac.jp/Tiaraday/yubimoji/demo.html
12) Namatame, M., et al.: "Letting Every Pupil Learn Japanese Hand Alphabets with Visual Interfaces", CHI 2004 Late Breaking Results Paper, (2004) 1363-1366