Dialog Support for Totally Deafblind Persons by Conveying Back-channels through Vibration

Akira Okamoto1, Misako Nambu2, Shigeki Miyoshi3, and Masatsugu Sakajiri4

1 Tsukuba University of Technology, 4-12-7 Kasuga, Tsukuba, Ibaraki, 305-8521, Japan.
e-mail: okamoto@k.tsukuba-tech.ac.jp, Phone: 029-858-9583, FAX: 029-858-9583
2 Future University-Hakodate, 3, 4Tsukuba University of Technology

Abstract: We conducted an experiment using a vibration system using which a totally deafblind participant, communicating through an interpreter, could directly receive back-channel feedback from a conversational partner. Statistical analysis of the interaction structure showed that the back-channel vibration increased the participant’s turn-taking ability and interactivity during the communication. Moreover, from a qualitative analysis of the video data, we found that the experience of receiving back-channel feedback caused a change in his haptic behavior with respect to that during normal communication. Our results indicate that the low mental workload and learnability of this vibration system allows deafblind persons to improve their communication. In parallel with the experiment using a vibrator, we conducted a questionnaire survey among the interpreters, questioning them on their interpretation behavior.
Keywords: Back-channel, Communication, Deafblind, Mental Workload, Vibration Information

1 Introduction
 Totally deafblind* persons mainly communicate with others by haptic means such as finger Braille and tactile signs, and through interpreters. Some of them communicate by voice, if they can speak. The amount of information that they can receive by haptic means is limited, and the information conveyed tends to be fractionated. In our observations of their communication through interpreters on several occasions, there were cases in which the interpreters could not convey much of the environmental information. In particular, back-channel information, like nods, was not sufficiently conveyed. Thus, they had some difficulty in acquiring sufficient information about the situation and the communicative behavior of their conversational partners. This often made them feel left out of the conversation.
 In our previous study1, 2, we developed and evaluated a vibration system that is activated by sensing a partner’s utterance. Although this system allows a deafblind person to be aware of the existence of their partner’s utterance, it does not, by design, convey the meaning of the utterance. We observed that the vibration system caused some changes in a deafblind participant’s physical movement and conversation rhythm. For instance, shortly after putting on the vibrator, the participant’s movements (e.g., gestures and signs) reduced temporarily. However, as he continued the conversation with the vibrator, his movements gradually returned to their normal frequency. These observations suggest two important issues for assistive technology: mental workload and learnability. Frequent vibrations from this system seemed to interfere with the participants’ haptic behavior and produced a large mental workload as they tried to guess the intended meaning. On the other hand, it seemed that the deafblind participants might be able to accept the vibrations as helpful information to improve their communication.
 The following experiment, therefore, focused on conveying back-channel information by vibrations. Back-channel feedback conveys simple but effective information about the partner’s responses, and does not require the participant to guess the meaning from the context. Because such back-channel vibrations might be easy-to-understand and require a small mental workload, they might have the potential to encourage active involvement of the participant in communication.
 In addition, we will also briefly discuss the results of the questionnaire survey for deafblind interpreters, with questions pertaining to their interpretation behavior. Although the interpreters were trying to convey as much back-channel information as possible, they were not satisfied with their own interpretation performance.
* Although the definition of “deafblind” includes not only totally deafblind but also the hard of hearing and low vision in general, we focus on totally deafblind persons in this paper, hereinafter referred to as just “deafblind.”

2 Experiment
 To investigate the effect of our back-channel vibration system on communication, we conducted an experiment with a male deafblind participant (in his mid-fifties, congenitally deaf, with acquired blindness, and without utterance). He was asked to talk freely with one of the experimenters through an interpreter in three sessions. In sessions 1 and 3, the vibrator was attached to an unobtrusive area of his left arm, and he could receive the vibration information as back-channel feedback from his partner. In session 2, the vibrator was removed. As he received information through tactile sign language and responded in normal sign language, he could not receive information from his partner while simultaneously responding in Session 2.
 In this experiment, the vibrator was operated manually by another experimenter, who activated it to convey the back-channel information from the partner.

3 Results and Discussion
 A multi-method approach was used to examine whether, and how, back-channel vibrations affect the interaction process for a deafblind participant, as described above.
3.1 Statistical Analysis of Interaction Structure
 To investigate the conversation’s rhythm, we analyzed indicators of the interaction structure in sessions 1–3.
 In sessions 1 and 3 (with back-channel vibrations), the time rate of the participant’s turn was significantly lower than in session 2 (X2(2) = 43.15, p<.01). Similarly, the duration time per turn in sessions 1 and 3 tended to be shorter than in session 2 (F(2) = 2.89, p<.10). There was no significant difference between sessions 1 and 3 for either indicator.
 In his usual communication through an interpreter, the participant is likely to carry on a one-sided conversation. Therefore, these results suggest that back-channel feedback by vibration has the advantage of increasing the interactivity and encouraging the active involvement of the deaf-blind participant. Moreover, the low mental workload and high learnability of the back-channel vibration system can be considered to be the key factors for improving communication.
3.2 Qualitative Analysis
 Focusing on the participant’s movement, a qualitative analysis of the video data was performed to detect the impact of the back-channel vibration, and we found two interesting cases.
 In the first case, after a few minutes of wearing the back-channel vibrator for the first time (Session 1), the participant nodded heartily toward the vibrator attached to his arm (Fig. 1, left). He had received a simple explanation of its function. However, this case showed that he truly understood and accepted the vibration as available information at this moment.
 Another case was found in session 2 (without the vibrator). He tried two-way communication using haptic means: delivering a message with his right hand, while simultaneously extending his left hand toward the interpreter seeking a response (Fig. 1, right). This challenge seems to have been evoked by his experience receiving the back-channel information in session 1. We can see this change in haptic behavior as a positive reaction to the system.
 Additionally, in an interview after the experiment, the participant pointed out that the back-channel vibration could be easily understood.

Fig. 1. Head nodding toward the vibrator attached to his arm (left) and reaching for a checking reaction while simultaneously delivering a message by sign (right).

4 Result of Questionnaire Survey
 In parallel with the experiment using a vibrator, we conducted a questionnaire survey for interpreters asking them about their interpretation behavior. Although our analysis of the questionnaire survey data is not yet complete, some of the results of the quantitative analysis are shown here.
4.1 Questionnaire Subjects and Procedure
 Skilled interpreters for the deafblind were asked to fill out the questionnaire. To gather as many subjects as possible, the years of experience of the interpreters and their qualifications were not considered.
 The questionnaire was distributed to 40 interpreters at an exchange meeting of deafblind persons held in Tokyo. Prior to distributing the questionnaire, the purpose of our research was explained. The same questionnaire was also sent to another 5 interpreters by e-mail.
4.2 Contents of the Questionnaire
 The following questions were included on the questionnaire.
(1) Reality of interpretation
 (1A) Most frequent situation of interpretation: select three from (a) Dialog with sighted-hearing person, (b) Conversation with multiple persons such as in a conference, (c) Lecture, and (d) Dialog with another deafblind person.
 (1B) Method of interpretation: select three from (a) Finger Braille, (b) Braille, (c) Tactile sign, (d) Sign for limited vision, (d) Finger signed KANA, (e) Hand writing on palm, (e) Writing messages, (f) Voice, and (g) Other.
(2) Information to be conveyed
 (2A) Examples of information attempted to convey and information attempted but failed to convey.
 (2B) Degree of successful interpretation: select one from (a) Almost all, (b) About 3/4, and (c) Below 1/2.
(3) Effort to convey back-channels
 (3A) During interpretation: Select one among (a) Try to convey everything, (b) Depends on subject matter, and (c) Do not convey.
 (3B) During personal conversation with deafblind person: Select one from (a) Try to convey everything, (b) Depends on subject matter, and (c) Do not convey.
4.3 Brief Report on Questionnaire Analysis
 Twenty two (22) answers were collected (17 by post, 1 by fax, and 4 by e-mail).
(1A) Major situations of interpretation:
 (b) Conversation with multiple persons such as at a conference (6 responses)
 (a) Dialog with sighted-hearing person (5)
 (c) Lecture (4)
(1B) Major methods of interpretation:
 (c) Tactile sign (11 responses)
 (a) Finger Braille (5)
(2A) The results will be reported after the qualitative analysis is completed.
(2B) Interpreters ratings of their successful interpretations:
 (c) Below 1/2 (13 responses)
 (b) About 3/4 (6)
 (a) Almost all (2)
(3A) Effort to convey back-channels at interpretation:
 (a) Try to convey everything (13 responses)
 (b) Depends on subject matter (8) (3B) Effort to convey back-channels during personal conversation with deafblind person:
 (a) Try to convey everything (17 responses)
 (b) Depends on subject matter (3)
 In relation to the situations for interpretation, it is natural that interpreters are frequently asked to interpret at conferences or lectures, but from (1A), the importance of interpretation during personal conversation is recognized.
 The results for (1B) show that the two major methods of interpretation are tactile sign language and finger Braille. This is because most of the questionnaire subjects were skilled interpreters. In our observation of their communication in other situations, several other methods of communication are used by non-skilled supporters. However, tactile sign language and finger Braille are also the major methods for their interpretation.
 As seen in (2B), the interpreters feel that their interpretation is insufficient. There may be several reasons for this fact. We expect that the qualitative analysis of (2A) will reveal some of the factors.
 In relation to conveying the back-channels, the results of (3A) and (3B) show that there is a difference in the interpreter’s efforts between personal conversation and an interpreting assignment. It seems that in interpretation, the interpreters think that conveying the contents of the information is the most important and that back-channels are sometimes regarded as low priority information, although they are aware of their importance. We will survey this topic further in the future.

5 Conclusion and Future Tasks
 Our results from both statistical and qualitative analyses indicate that the back-channel vibrations affected the communication of the deafblind participant. There was a significant increase in the interactivity and his haptic behavior became more actively involved in the communication. The smaller mental workload and learnability of this system are considered to be key factors in effective assistance.
 To verify this conclusion, we will conduct a more detailed analysis, focusing on both the interaction structure and content. We would also like to make the benefits of our system available in everyday activities and conduct detailed observations of the daily-life situations of deafblind persons.
 We will complete the qualitative analysis of the questionnaire survey. We will also continue with observations of deafblind persons’ communication.

References
1. Misako Nambu, Akira Okamoto, Shigeki Miyoshi, Masatsugu Sakajiri. Communication of Deaf-Blind Persons Using Vibrating Device: A Preliminary Experiment. IEICE Technical report, Well-being Information Technology (WIT2006-36), pp.27-30, (in Japanese)
2. Misako Nambu, Akira Okamoto, Shigeki Miyoshi, Masatsugu Sakajiri. Dialog Support for Deafblind Persons by Conveying Backchannels through Vibration. Computers Helping People with Special Needs, 11th International Conference, ICCHP 2008 Proceedings, pp.686-689