Evaluation Pattern Editing Tool for Visually Impaired Students: Developing Application Software for Wearable Haptic Tactile Interface 

YABU Ken-ichiro1,2), SAKAJIRI Masatsugu2), ONISHI Junji2), IFUKUBE Tohru1) 
1)Institute of Gerontology, the University of Tokyo,
	7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan
	E-mail: yabu@human.iog.u-tokyo.ac.jp, Tel/Fax: +81-3-5841-1524
	2)Department of Information Technology, Faculty of Health Sciences,
	Tsukuba University of Technology
	
Abstract: TAJODA II is a device developed as a tactile aid for hearing and/or visually impaired people. The device includes a small camera, a small microphone, an accelerometer, and a tactile display. The tactile display transforms the information detected by sensors into vibration patterns. The tactile display also functions as a tactile input sensor. This bidirectional feature might allow bidirectional communication in order to assist deaf and blind people. TAJODA II operates based on an internal embedded software that can interpret original simple interpreter programs. By rewriting such programs from a personal computer with a USB, operations can be changed easily. Thus, an appropriate application software is required to improve TAJODA II so that it becomes a more helpful device. Subsequently, a blind student from our college group attempted to develop a pattern on a tactile display for the development of an application software. However, using the software for development with a screen reader was difficult for this student. Therefore, an evaluation pattern editing tool for visually impaired developers was developed. This paper describes the design concept of a pattern editing tool, and explains an approach for developing patterns by visually impaired college students for the purpose of research and education. Keywords: Deaf-blind, Mobile device, Tactile display, Visually impaired 

1. Introduction 
Mobile or wearable information devices, such as smartphones or tablet computers, have become widespread. The reasons behind the spread of these information devices are that they are wearable (or mobile), usable in real-time, and bidirectional (allow interactive communication). 
Various tactile aids have been proposed as sensory substitution devices for acquiring visual and sound information more interactively [1-8]. Optacon [3] is one of the most famous electromechanical sensory substitution devices. Moreover, �gTAJODA�h [4, 5] is a device that was developed in our laboratory and presents additional information tactually in conjunction with a text-to-speech reading system on a personal computer. Furthermore, a tactile vocoder has been developed [2, 6, 7] not only for the visually impaired, but also for deaf and blind people and the hearing-impaired. 
In addition, we developed TAJODA II as an aid for the hearing and/or visually impaired (Fig. 1). The functional details are described in a previous report [1]. TAJODA II is a device that includes a small camera, a small microphone, an accelerometer, and a tactile display, with a view to integrate the device with mobile devices such as smartphones. The tactile display consists of a 32-channel vibrator matrix. The vibrators can represent the information detected by sensors as vibration patterns. Furthermore, the tactile display also functions as a tactile input sensor. The bidirectional feature might allow bidirectional communication in order to assist deaf and blind people. These devices are wearable, usable in real-time, and bidirectional (interactive communication). 
By rewriting the internal interpreter program, the sensor to be used and the vibration pattern of a tactile display can be changed. Therefore, in order for this device to support hearing and/or visually impaired people appropriately, it is necessary to develop a suitable software application. Thus, the type of vibration pattern, speed, and direction for transmission of optimal information need to be investigated. However, because there are innumerable combinations, a method for optimizing experientially is obtained by a cycle of trial and error. 
Fig. 1. TAJODA II and tactile display. 
In general, because visually impaired people acquire information from Braille tactically, they have the ability of reading information on a tactile display compared to people having sight Therefore, when people have sight develop software for TAJODA II, this difference can become a problem when designing a tactile sense pattern. If visually impaired people could develop vibration patterns for the tactile display on TAJODA II, this problem might be avoided in the design phase. 
On the other hand, when a blind developer designs and develops something, there is often a barrier where sufficient development environment is not ready. A student with visual impairment who wants to develop a support device is a part of the laboratory of our group. However, because there is a part that depends on GUI, he has difficulty in using the software for development with support devices, such as a screen reader. 
Consequently, we developed an evaluation pattern editing tool with which a blind developer can design an oscillating pattern by using a screen reader and touching the tactile display on TAJODA II interactively. This paper describes the design concept for an evaluation pattern editing tool, and explains an approach for developing patterns by a visually impaired college student for the purpose of research and education. 
2. System configuration for evaluation pattern editing tool for visually impaired individual 
2.1 Tactile display and control system with personal computer 
The tactile display consists of a matrix of 32 small vibrator rods. The pins on the tactile display are arranged in the shape of a matrix at intervals of 2 mm. These pins are positioned in a 6�~6 square, and the pins at the four corners of the square are removed (right side of Fig. 1). Each pin vibrates at a frequency of 200 Hz, with a proper timing based on an internal software or commands from the personal computer connected by a USB cable. In order to prevent the finger from becoming desensitized to a stimulus, the vibration amplitude of the pins is modulated at approximately 60 Hz. 
The oscillating intensity is set up for each pin by using an internal software or a software from the personal computer. Moreover, there is a function where the vibration is controlled by the signal from the computer connected by USB. In this function, when a command of intensity of all 32 pins is transmitted to TAJODA II, an oscillating pattern is presented in real time on a tactile display. When the commands are sequentially transmitted at a constant interval, a pattern that moves on the tactile display is presented. 

Fig. 2. Evaluation pattern editing tool for visually impaired student. 
2.2 Evaluation pattern editing tool for visually impaired 
The evaluation pattern editing tool for the visually impaired is shown in Fig. 2. The tool consists of a text box used for editing the entire sequence data (Fig. 2-(1)), a grid view for inputting the control value of each of the 32 pins (Fig. 2-(2)), a grid view for reading the control value of the pin outputted to the current device (Fig. 2-(3)), and a button group that performs various types of control. In order for the standard accessibility function of Windows to work correctly, this tool was developed in the C# language, where various types of standard UI components have been prepared. 
2.1.1 Editing sequential pattern on text box 
The oscillating pattern that moves on a tactile display is edited in the text box for editing sequence data. The text box can be employed using a screen reader and a standard Braille display. The moving vibration pattern consists of some pattern frames transmitted continuously. The oscillating intensity data of one frame is written by comma-separated values in the 6�~6 matrix. (The variable of the four corners is ignored at the time of transmission.) The pin intensity is denoted by an integral value between 0.255 (e.g., Fig. 2). The line that starts with �g//�h divides the frame data. After connecting TAJODA II to a personal computer by using a USB cable and by clicking the �gconnect�h button, upon clicking the �gsend the continuous sequence�h button, the data of each frame are transmitted sequentially, and the moving pin pattern is presented on the tactile display. In addition, if the integral value is written after the character string �g//�h, the last frame continues with the written time interval. 



2.2.2 Editing pattern on a grid view 
A grid view makes it possible to edit only the data of the pattern of each pin in a frame. A grid view consists of 6�~6 matrices, and an integral value in the range 0.255 is inputted into each cell. After connecting TAJODA II with a personal computer by using a USB cable and by pressing the �gconnect�h button, upon pressing the �gsend�h button, the oscillating intensity value written in the grid view is transmitted to TAJODA II, and the pattern is immediately displayed on the tactile display. 
When the check box for �gtransmit changed pattern immediately�h is selected, after the value of the cell in the grid view is changed, the value is immediately transmitted to TAJODA II. The developer can edit a control value by touching a tactile display in real time and checking it. Pressing the �gappend the pattern in the grid view to the sequence�h button appends the value inputted into the grid view to the text in the text box described in the previous section by using CSV. 
The tool also has a function that shifts the entire data in the grid view. When the �gup,�h �gdown,�h �gright,�h and �gleft�h buttons are clicked, all the data in the grid view shifts in the direction indicated by the button. There is also a function where the sequence data for the pattern written in the grid view to scroll vertically and horizontally are appended to a text box. 


3. Preliminary experiment 
3.1 Method 
In order to investigate how a student can actually use the pattern editing tool described in the previous section, we conducted a simple experiment in cooperation with a visually impaired participant. The participant is a college student in his 20s who is congenitally blind. He uses Braille and a screen reader daily, and has acquired programming skills for using the screen reader and Braille display. 
The experiment was conducted in two steps. In the first step, the participant operated the pattern editing tool and TAJODA II based on the instructions of an experimenter with sight and grasped the operation method. In the second step, the experimenter only taught the student to create an image of the capital alphabet letter �gM;�h the student then continued the operation by using trial and error independently. 

3.2 Results and discussion 
An example of the pattern edited by the student in the second experimental step is shown in Fig. 3. The student saved the sequence data that this pattern scrolls vertically and horizontally for approximately 1 h after the experiment starts without any help. The first and second steps were performed in approximately 50 and 10 min, respectively, until the pattern was saved as a text file. The sequence data edited in the second step consists of 133 frames. 
The student expressed the following comments regarding the editing tool: 
. 	
It is better to read the first line of the grid view with �gthe 1st line,�h using the screen reader although the first line is read with �g0th�h. 

. 	
Although a shortcut key was not assigned to the button for appending scroll data, all the buttons should be assigned the shortcut key, e.g., �gShift + 1�h or �gShift + 2.�h 

. 	
If three areas, �gsequence data�h (Fig. 2-(1)), �gpattern creation�h (Fig. 2-(2)), �gthe current pattern�h (Fig. 2-(3)), and other control buttons are divided by tab control, the focus can move quickly to the point that is to be edited (the tabs are changed by pressing the �gCtrl+Tab�h key). 

The student expressed the following comments regarding the hardware of TAJODA II: 

. 	
A bigger tactile display (for example, 30�~30 pins) could allow application to game software. 

. 	
A thinner pin could raise the resolution. 




Fig. 3. Pattern of frame edited by student in experiment. 
As described above, with regard to the pattern editing device, many ideas for allowing smooth operation with a keyword were proposed. A  concrete shortcut key was proposed, as was the idea of substituting the user interface  with a tab control that allows smooth switching with the �gCtrl+Tab�h key. We believe that the student was able to  propose these ideas because he always operates personal computers by using only the keyboard. Some improvement points were proposed not only with regard to a pin pattern editing device, but also with regard  to hardware. We believe that the student was able to propose these ideas because he is a completely blind person and has some experience with software development. Although adding improvements to hardware directly is difficult for this student, his ideas can be included in the hardware design for the following version of this device. 
4. Conclusion This paper reported the design concept of an evaluation pattern editing tool for visually impaired developers, and explained an approach for developing patterns by a visually impaired college student for the purpose of research and education. In the preliminary experiment that used the pattern editing tool, the student could edit the pattern data for a capital letter �gM�h that scrolls vertically and horizontally for approximately 1 h after starting the first instruction on the manipulation of the tool. Subsequently, he was able to propose some improvement points for the hardware and software. From the perspective of the development of a support device for the visually impaired, if a blind person develops the device and software application using the sensation of his/her finger, the problem of the difference in the ability of reading information from a tactile sensor between persons with sight and the visually impaired can be avoided. In addition, because blind developers can imagine a scene where such support devices are used in their daily life, they might be able to develop more concrete and suitable content for these devices. In many cases, after a person with sight develops a device, the visually impaired participate in experiments, and the device is improved based on an  investigation of the usability or required functions. Device development by a blind person can shorten the term for the cycle of trial and error, more trials can be performed, and a more effective device can be produced. On  the  other hand, from  the  perspective  of the  research  and  education of hearing and/or visually impaired  students, the case often exists where developing a device is difficult for such students because no available  development environment exists (to the best of our knowledge), regardless of the students�f development skills. For example, if the control software for a device is not read correctly with a screen reader, a barrier is formed. In  such  cases,  the  development  of  a  system  that  permits the  use  of  a  screen  reader  and  the  elaboration  of  another  development environment tool that can be used with the screen reader can remove the barrier for students to develop a  device. By  preparing  a  suitable  development  environment,  students can develop new devices by themselves.  If possible, in cooperation with teaching staff or laboratory colleagues, students should improve the original  development environment. The evaluation pattern editing tool described in this paper is an example of a tool used to evaluate the pattern of 
the tactile display on TAJODA II, It is an approach to developing new application software for the tactile display. The experiment showed that some improvement points are required for the editing tool. Improvement of the development tool is important for improving TAJODA II. We will continue evaluating and improving this editing tool to add enhancements to the hardware device. 

5. Acknowledgements 
This work was partially supported by JSPS KAKENHI Grant Numbers 26280070, 15K04540, 15K01015, and 26750219. 
We are grateful for the assistance of Atsushi Katagiri (Push-Pop Co., Ltd.), who made an important contribution to the development of the TAJODA II device. 

References 
[1]		Yabu K., Sakajiri M., Ifukube T. Development of a wearable haptic tactile interface as an aid for the hearing and/or visually impaired. NTUT Education of Disabilities. 2014; 13: p.5.12. 
[2]		Ifukube T., Yoshinoto C. A sono-tactile deaf-aid made of piezoelectric vibrator array. Journal of the Acoustical Society of Japan. 1974; 30(8): p.461.462. 
[3]		Goldish L.H., Taylor, H.E. The Optacon: A valuable device for blind persons. New Outlook for the Blind. 1974; 68(2): p.49.56. 
[4]		Asakawa C., Takagi H., Ino S., et al. TAJODA: Proposed tactile and jog dial interface for the blind. IEICE Transactions on Information and Systems. 2004; 87(6): p.1405.1414. 
[5]		Wada C., Shoji H., Ifukube T. Development and evaluation of tactile display for a tactile vocoder. Technology and Disability. 1999; 11(3): p.151.159. 
[6] Sakajiri M., Miyoshi S., Nakamura K., et al. Accuracy of voice pitch control in singing using tactile voice pitch feedback display. 2013 IEEE International Conference on Systems, Man, and Cybernetics (SMC). 2013; p.4201.4206. 
[7] Sakajiri M., Miyoshi S., Nakamura K., et al. Development of voice pitch control system using two dimensional tactile display for the deafblind or the hearing impaired persons. NTUT Education of Disabilities. 2011; 9: p.9.12. 
[8]		Kaibai Y., Yao H., Ueda K., et al. Development of an interactive wearable tactile device for communication support. IEICE Technical Report. Welfare Information Technology. 2011; 111(58): p.47.52. (Japanese)