Development of a Wearable Haptic Tactile Interface as an Aid for the Hearing and/or Visually Impaired YABU Ken-ichiro1,2), SAKAJIRI Masatsugu2), IFUKUBE Tohru1) 1)Institute of Gerontology, the University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan E-mail: yabu@human.iog.u-tokyo.ac.jp, Tel/Fax: +81-3-5841-1524 2)Department of Information Technology, Faculty of Health Sciences, Tsukuba University of Technology Abstract: A wearable haptic tactile interface, TAJODA II, was designed as an aid for the hearing and/or visually impaired. The device includes a small camera, small microphone, accelerometer, and tactile display so that it can be integrated with a mobile device such as a smartphone. The whole device measures 69~115~29 mm and weighs 160 g excluding the batteries (two AAA). The tactile display consists of a 32-channel vibrator matrix, and has the function of representing images or sounds detected by the camera or the microphone as vibration patterns on the fingertip pad. Since the tactile display also functions as a tactile sensor, tactile information obtained by striking or tracing the display is presented on another receiverfs tactile display, enabling it to be used also for deaf-blind people. This paper describes the design concept and the basic functions of the tactile interface, and explains how the interface is useful for the blind based on the results of preliminary evaluation experiments carried out by a blind person. Keywords: Deaf-blind, Mobile device, Tactile display, Visually impaired 1.Background and introduction Braille and tactile graphics are used by visually impaired people and deaf-blind people to read information represented with diagrams or characters. Various electromechanical devices that can redraw these diagrams and figures by computer, such as Braille displays and tactile displays, are available. These devices are mainly used for perceiving the data on personal computers. Character data and image data are converted into Braille or digitized pictures on a personal computer, and are displayed on a device. However, visually impaired people and deaf-blind people need to be able also to use information other than characters or simple diagrams on personal computers. Various tactile aids have been proposed as a sensory substitution device for acquiring visual and sound information more interactively [1-7]. For example, the Optacon [2] is one of the most famous electromechanical sensory substitution devices. It is a portable device that presents the image acquired from the optical sensor in the form of vibrations of a 24~6 matrix of tiny metal rods. The gTAJODAh [3, 4], shown in Fig. 1 (a), was developed at our laboratory. This device presents additional information tactually in conjunction with the text-to-speech reading system on a personal computer. We also developed the tactile vocoder [1, 5, 6], shown in Fig. 1 (b), as we consider that not only visually impaired people but also deaf-blind people and the hearing-impaired hear sounds. TAJODA presents information, including punctuation, bold characters, italic characters, colors, etc. as an oscillating pattern of the vibrator matrix at the fingertip, along with the voice of the text-to-speech reading system. Moreover, the paragraph to be read out and the reading speed can be controlled in real time with a dial interface. Fig. 1 (a) gTAJODAh and (b) a gTactile Vocoderh On the other hand, mobile communication devices such as the smartphone have spread quickly in recent years. Against this background, we considered that mobile interfaces that can transmit images, sound, and hand motions to the tactile sense of a fingertip through a tactile display are very helpful, supporting the sensation and communication of visually impaired and deaf-blind people, and elderly people whose visual and hearing functions have declined, etc. These devices have the features of being wearable, real-time, and bidirectional (interactive communication) [7]. Thus, we developed a portable tactile display device, TAJODA II (Fig. 2), which includes a small camera, small microphone, accelerometer, and tactile display, with a view to integration with a mobile device such as a smartphone. Vibrators of this tactile display can generate oscillating stimulus that may be fully perceived by several volts using a specially developed piezoelectric element. The 32 vibrating pins on the tactile display are arranged in the shape of a matrix at intervals of 2 mm. The tactile display also functions as a tactile sensor, enabling the detection of information with a tap or stroke. Fig. 2 TAJODA‡Uand the Tactile Display. The TAJODA II has the function of representing images or sounds captured by the camera or the microphone as vibration pattern information, and the function of representing information from hand or finger motion, which is detected with the tactile sensor and accelerometer built into another transmitting side device. Since the device combines the gOptaconh, a reading support device for visually impaired people developed in the past, and the gTactile Vocoderh, a hearing support device that we developed for hearing-impaired people, and also enables communication to be made by touch, it can also assist deaf-blind people. Thus, this tactile display has three features: gwearableh, gin real timeh, and ginteractive.h This paper describes the design concept and the basic functions of this new device, and reports the results of a preliminary perception experiment using this device. 2. System configuration 2.1. System-wide configuration The TAJODA II is a rectangular parallelepiped measuring 69~115~29 mm, and it has a tactile display in a 22-mm square region at the upper central part of the surface. The device weighs 160 g excluding the batteries (two AAA), and costs slightly less than 200,000 yen including the software and PC interface, although it is built to order. The user places a fingertip on the display and perceives the vibration through the cutaneous sensation in the finger. There is an image sensor on the back, and a small microphone, an accelerometer, pronunciation sounder, Bluetooth module, and USB connector inside the device. Moreover, as described later, the pins of the tactile display also become pressure-sensitive sensors. When the tactile display is lightly tapped with a finger, the information on the tapping force is input. Operation of these sensor modules is controlled by a field-programmable gate array (FPGA) and a microcomputer. The device uses just two ordinary AAA batteries, and it runs alone based on the internal embedded software that can interpret original simple interpreter programs written with text data. By rewriting the interpreter programs and transmitting them from a personal computer by USB, the operations of the device can be changed easily. Moreover, through the control software on the PC, the state of each sensor can be displayed (Fig. 3) and the operating parameters can be set. The functions of each sensor module and the control software are described below. Fig. 3 Control Software of TAJODA‡Uon a Personal Computer 2.2. Tactile display The tactile display consists of a matrix of 32 small vibrator rods. These pins are positioned in a 6~6 square, removing the pins at the four corners. Each pin vibrates at a frequency of 200 Hz, with proper timing based on internal software. To prevent the finger from becoming desensitized to a stimulus, the vibration amplitude of the pins is modulated at about 60 Hz. The oscillating intensity is set up for each pin using the internal software or the software on the personal computer, and is controlled by a microcomputer and FPGA. These pins are operated by interpreter programs as described above. The pins are vibrated by bimorph-type piezoelectric elements developed for this device. Piezoelectric elements used for our previous version of the experimental model needed AC 56 V to produce an amplitude of 110 micrometers [4]. As a result of trial-and-error with the engineers involved in this research, we developed a laminated type of bimorph piezoelectric element that vibrates with an amplitude of more than 60 micrometers only by AC 5 V (Fig. 4). Furthermore, since the resonance frequency was set at a high frequency of 1200 Hz, power is transmitted well, without the vibration of a pin becoming weak when a finger is applied. The pins on the actual device vibrate with a square-wave voltage of about 12 V. Furthermore, since the vibrator on the tactile display consists of the piezoelectric element, the changes to an element are converted into a voltage, contrary to a voltage being used to vibrate an element. Using this principle, the signal produced by tapping lightly on the tactile display is detectable. For example as a concept, using the Bluetooth module built in two sets of TAJODA II, when the tactile display of one device is lightly tapped, the tactile display of the other side can vibrate. This can be used as tactual communication. In the control software on the personal computer, the user can check the position of the pin that is vibrating, and the pressure-sensitive information on a pin in real time. The upper left in Fig. 3 shows the screen displaying gLh as the pin that is vibrating. Regarding the information input by tapping a pin with a finger, the force is shown by the size of each circle with a spatial resolution which divides the tactile display into four (Fig.5 (a)). Fig. 4 Peak to peak displacement of vibration of tip of piezoelectric elements excited by AC 5V Fig. 5 Information of each sensor displayed on the screen of the control software on a personal computer. 2.3. Imaging sensor An image sensor is built into the back of the device in order to represent visual images on the tactile display. Users can perceive brightness and patterns as vibrations by turning the camera on the back of the device towards an object, with their finger on the tactile display. Although the image sensor itself has a maximum performance of 1280~1024 pixels and 15 fps, since there are restrictions on the size of a tactile display and the distance between pins, the image acquired from a sensor is controlled to about 128~96 pixels and 10 fps, in order to reduce the generation of excessive data. The software further compresses the obtained image data, cuts the pixels to the right and left of it, and deals with it as a 48~48-pixel image. These processes are performed by the built-in FPGA and microcomputer. Three image processing algorithms are installed in the present version of the TAJODA II. The first algorithm is used to display the brightness from the imaging sensor on the tactile display (lower left of Fig. 5 (b)). That is, when the pixel of a certain point acquired from the imaging sensor is brighter than the threshold value, the pin corresponding to that point vibrates. The second algorithm is equivalent to the reverse display of the first algorithm (lower right of Fig. 5 (b)). That is, when the pixel of a certain point acquired from the imaging sensor is darker than the threshold value, the pin corresponding to that point vibrates. The purpose of the first and the second algorithm is to make it possible to recognize simply the brightness of the color of the object toward which the imaging sensor was turned, and the brightness of a view by a fingertip. The third algorithm extracts the edge of the object contained in the image and displays it on the tactile display (upper right of Fig. 5 (b)). This algorithm is for cases where, for example, a user recognizes the character and figure that were printed on paper as vibrations on the fingertip, when the user turns the imaging sensor toward himself or herself. The threshold values and sensitivity of the three algorithms described above are controllable by the control software on the PC, and an internal interpreter program. The zoom level of the imaging sensor is also controllable. We will continue to improve the image processing algorithms by repeating the experiments and identifying the most suitable threshold values, sensitivity, and zoom level. 2.4. Accelerometer The built-in accelerometer is used to detect when the device is being moved or slanted by the userfs hand. The signal of the accelerometer can always be acquired if the power supply of the device is on. Detectable directions are the three dimensions of back and forth, right and left, and up and down. Acceleration is indicated on the screen of the control software on the personal computer, as shown in Fig. 5 (c). Acceleration is shown by the size and position of a circle. In the case of horizontal acceleration, the circle moves sideways and up and down. For vertical acceleration, the size of the circle changes according to the acceleration. 2.5. Microphone The small built-in microphone enables deaf-blind people and the hearing-impaired to be made aware of nearby voices through a warning sound in the form of a vibration of the tactile display. The sound acquired by the microphone is 12-kHz stereo sound. The sound is processed by six independent digital bandpass filters. The center frequency of each bandpass filter is 50.6,000 Hz, which can be controlled by the preset value of the control software, or by the interpreter program input by the user. Users can set it up as other sensors, checking the audio state on the screen of the control program on the personal computer. However, since the sound that occurs from the vibration of a pin is picked up by the microphone and causes howling while the pin vibrates, a countermeasure is required and thus the algorithm is being improved. 2.6. Font display function In addition to the function of each sensor of the device described above, the TAJODA II can present optional patterns by vibrating pins using the control software or instructions from the interpreter program. A font system is also installed and alphabet letters can be displayed on the tactile display according to the control software or interpreter program. The scroll speed and oscillating intensity of the alphabet letters on the tactile display can be changed, and custom-made fonts can also be set up. 3. Preliminary perception experiment After completing an experimental model of the interface, in order to confirm what information is transmitted by the representation on the tactile display, we conducted a simple recognition experiment in cooperation with a visually-impaired participant. The participant is a man in his 40s who is totally blind (acquired impairment). He uses Braille on a daily basis and had previously received training for using the Optacon. Fig. 6 A preliminary trial experiment. 3.1. Method The participant operated the TAJODAII according to the experimenter's instructions, and the target stimulus was presented on the tactile display in stand-alone mode. The stimulus was repeatedly presented at a constant interval until the experiment was finished. The participant placed his index finger on the tactile display of the TAJODAII (Fig. 6), and orally described what was presented on the tactile display in a dialog with the experimenter. The presented stimuli were the following two patterns. The scene of the experiment was recorded on video and analyzed. (1) Line moving sideways and up and down As shown in Fig. 7, the pin located on a horizontal line vibrated and the position moved up and down. Next, the pin located on a vertical line vibrated and the position moved to the right and left. The stimulus was presented repeatedly at a speed of about 10 seconds (2.5 s ~ 4) in one cycle in which the vibration of the pin moves sideways and up and down. The oscillating pattern of the pin was repeated until the participant recognized the pattern. While the vibrating line of pins moved from a certain line to the following line, a phase was added where two lines of pins vibrated, so that the line located at half the distance between pins was expressed, as shown in Fig. 7. (2) Alphabet letters One phrase of gHELLO WORLDh was displayed in capital letters on the tactile display. The characters displayed on a tactile display in the stimulus are shown in Fig. 8. Each character appeared from the right of the tactile display, and scrolled toward the left at about 2.5 seconds per character. There was a pause of about 2.5 seconds between gHELLOh and gWORLDh so that the space might be represented. The display of one whole gHELLO WORLDh phrase was also repeatedly presented at a constant interval on the tactile display until the experiment was completed. At the start of the experiment for the participant, there was either no information that a character was displayed, or the same word came out repeatedly, or a certain phrase was displayed. Fig. 8 Characters displayed on a tactile display in stimulus gHELLO WORLDh 3.2. Results In the experiment, the first stimulus (1) was presented, and then the second stimulus (2) was presented. The participant recognized the pattern of the displayed stimulus (1) immediately, and responded gtop to bottomh, gbottom to toph, gright to lefth, gleft to righth. Therefore, the experiment for stimulus (1) was finished in about 10 seconds. The experiment for stimulus (2) was conducted for about 5 minutes. Although the phrase gHELLO WORLDh was not recognized within that time, the participant correctly guessed some of the characters later on in the experiment. First, the subject recognized that more characters came out, in about 15 seconds. After incorrectly guessing characters such as gUh, gVh, etc., he correctly recognized gLh 10 seconds after the experiment started. In the early stage, although he confused gVh with gWh, he eventually recognized it as gWh. He had noticed that the same pattern came out repeatedly about 1 minute and 15 seconds after the experiment started. He responded with the character string gP, E, L, L, O, W, O, S, L, Dh at the end of the experiment. In terms of each character, he did not recognize gHh at all during the experiment. He appeared to recognize gEh, gLh, and gWh almost certainly about 1 minute after the stimulus was presented. He recognized gRh only twice during the experiment. He appeared to distinguish gOh and gDh certainly at the end of the experiment although he had confused them at the beginning of the experiment. 3.3. Discussion Regarding stimulus (1) for the line moving sideways and up and down displayed on the TAJODAII, the participant recognized it immediately. In a simple experiment conducted later, all of the subjects including persons who were not physically handicapped answered that they recognized it easily. Therefore, it is considered that the developed tactile display is sufficiently effective for displaying such simple information. In talking with the participant after the stimulus (2) experiment, when he was told that the displayed characters formed the phrase gHELLO WORLDh, he quickly agreed and said, gI had followed only the appearance of the written lettersh with admiration. So, if the participant had been told beforehand that the displayed character string was one certain phrase, he might have correctly guessed that it was gHELLO WORLDh. After the experiment, when it was revealed that the first character was gHh, the participant touched the pins again and agreed quickly. As to the reason why gHh was not recognized, he answered that the lower right of the gHh could not be clearly recognized. This reason might be related to the participant's finger sensations, or to a deviation in the vibration of the pins on the tactile display. This could be improved by adjusting the oscillating amplitude on the tactile display so that it is perceived with the same strength for every pin. When the participant was told what was displayed on the tactile display, he seemed to realize the object very well as the stimulus of the tactile sense on a display. This indicates that he might be able to recognize almost all characters, if mapping of the vibrations on the tactile display and the characters were learned and practiced beforehand. To improve recognition, it is necessary to investigate a suitable font, scroll speed and amplitude in the future. 4. Conclusions This paper reported the design concept and the basic functions of gTAJODA IIh developed as a wearable interface device that works in real time and is interactive. Based on the preliminary perception experiment using the device, we investigated the extent to which experiment participants correctly recognized alphabet letters on the tactile display. The experiment results were video-recorded and the recognition speeds as well as the correct recognition were analyzed. From the analysis of the video, it was found that the blind user could immediately recognize the vibrating points sweeping along the display at 2.5 seconds each way. Furthermore, in the other simple experiments, all the participants including persons with normal eyesight answered that they could easily recognize these stimuli. When the character string gHELLO WORLDh was presented on the tactile display and moved like an electrical scoreboard at about 2.5 seconds per character, a totally blind participant correctly recognized many of the characters in real time without any training. Since the tactile display, CCD imaging sensor, small microphone, accelerometer the tactile sensor are mounted in the TAJODAII, various applications would be possible by utilizing these sensors. For example, by coordinating the interface with text-to-speech reading software, it would be possible to transform rich text information and simple figures that cannot be expressed only in voice into tactile information. Furthermore, since the tactile display also works as a tactile senor matrix, the speed of the synthesized voices could be controlled by tactile information sensed by tactile display. Moreover, the position and direction of the environmental brightness and sound source can be appropriately presented on the tactile display by moving the camera or the microphone of the device. Therefore, the device might also be effective for the deaf-blind as well as the visually impaired and the hearing-impaired for detecting danger in their environment while walking. In order to realize the above, it is necessary to conduct many evaluation experiments under various conditions, and to identify optimal parameters such as the information scroll speed and the vibration intensity. Finally, we hope that an increasing number of researchers, caregivers for disabled people, and other parties conducting evaluation experiments will use our tactile display. Acknowledgement This work was supported by JSPS KAKENHI Grant Number 26280070: Grant-in-Aid for Scientific Research (B) gPractical study of wearable haptic interface to support the sensory impaired and communication disordersh. We are particularly grateful for the assistance of Atsushi Katagiri (Push-Pop Co. Ltd.) who made an important contribution to the development of the TAJODA II device. References [1] Ifukube T., Yoshinoto C. A sono-tactile deaf-aid made of piezoelectric vibrator array. Journal of the Acoustical Society of Japan, 1974; 30(8): 461.462. [2] Goldish L.H., Taylor, H.E., The Optacon: a valuable device for blind persons. New Outlook for the Blind. 1974; 68(2): 49.56. [3] Asakawa C., Takagi H., Ino S., et al. TAJODA: Proposed tactile and jog dial interface for the blind. IEICE Transactions on Information and Systems, 2004; 87(6): 1405.1414. [4] Wada C., Shoji H., Ifukube T. Development and evaluation of tactile display for a tactile vocoder. Technology and Disability, 1999; 11(3): 151.159. [5] Sakajiri M., Miyoshi S., Nakamura K., et al. Accuracy of voice pitch control in singing using tactile voice pitch feedback display. 2013 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2013; 4201.4206. [6] Sakajiri M., Miyoshi S., Nakamura K., et al. Development of voice pitch control system using two dimensional tactile display for the deafblind or the hearing impaired persons. NTUT Education of Disabilities, 2011; 9: 9.12. [7] Kaibai Y., Yao H., Ueda K., et al. Development of an interactive wearable tactile device for communication support. IEICE Technical Report. Welfare Information Technology, 2011; 111(58): 47.52. (in Japanese)