User Evaluation of an Electronic Malaysian Sign Language Dictionary : e-Sign Dictionary

A software system needs to be maintained through continuous enhancements and improvements. To understand the types of enhancements to be made, a user evaluation exercise is often conducted to determine the system’s weaknesses and limitations. This paper reports on the outcomes of a user evaluation survey carried out on an electronic Malaysian Sign Language Dictionary, e-Sign Dictionary. The evaluation was conducted using a questionnaire survey focusing on the functions and features of e-Sign Dictionary. A total of 45 respondents comprising deaf school teacher, deaf students, and the general public participated in the user evaluation. The results show that out of 45 respondents, 36 (81%) respondents rated e-Sign Dictionary as a good or very good system. The results of the user evaluation would be used as guidelines to enhance and improve the functions and features of e-Sign Dictionary.

e-Sign Dictionary is an electronic sign language dictionary developed for the use of the deaf community and the general public in Malaysia.It is a Windows-based system that provides instructions in English, Malay, and Chinese language.The dictionary has 500 words together with animation to illustrate the signing of each word.Each word has: explanation; illustration of its meaning with a picture (if available); synonym; antonym; and a sample sentence to illustrate its usage (Ow & Lim, 2008).The system can perform eight main functions that include allowing a system administrator to maintain the dictionary database; and users to learn Malaysian sign language with ease.Currently, the dictionary is being used by teachers and deaf students of a deaf school in Malaysia.
To evaluate the functionalities and features of e-Sign Dictionary, it is important to gather feedback and comments from the users, especially, the teachers and deaf students of the deaf school, and the general public.This paper presents the outcomes of user evaluation on e-Sign Dictionary.The feedback and comments from the users would be used as guidelines to make enhancements and improvement to the dictionary.

Design of user evaluation form
A few methods can be used to evaluate e-Sign Dictionary.These include: interviews which can be carried out via telephone or face-to-face; survey which can be conducted using paper-based questionnaire through the mail, or through the telephone, or through the Internet (Web-based survey) (Bourque & Fielder, 2003a;Bourque & Fielder, 2003b; DIIA assessment consulting services, 2007a); and observing the user using the system.Each method has its pros and cons.Among these methods, the survey is considered to be the most suitable as it can be used to obtain a large amount of users' feedback and comments quickly.It is also economical to conduct a survey as compared to the other methods (Fink, 2006;DIIA assessment consulting services, 2007b).
Before conducting the user evaluation survey, an evaluation form was designed.The form consists of two sections.The first section gathers the user's profile, and the second section gathers the user's evaluation of e-Sign Dictionary.The evaluation section is divided into Part A and Part B. Part A contains questions pertaining to the functions and features of e-Sign Dictionary, while Part B allows the user to suggest improvements to further enhance the functionality of the dictionary (Appendix I).A pilot test using the evaluation form was conducted.It involved 10 users.The results of the pilot test show that the questions are easy to understand and answer.Hence, the evaluation form was used in the evaluation process.

User evaluation process
The evaluation process involved two groups of participants.The first group comprised deaf school teacher, deaf students, and parents of a deaf student.As e-Sign Dictionary was donated to the deaf school a year ago, the teacher and the deaf students have been using it for three months or longer.The second group comprised the general public who were chosen at random during an exhibition.They were shown how to use e-Sign Dictionary to understand its functions and features, before participating in the evaluation process.The first evaluation was carried out for one week in the deaf school, and the second evaluation was carried out from 24-26 June 2008 during an exhibition.Altogether, 15 sets of evaluation forms were collected from the deaf school, and 30 sets were collected from the general public.Hence, data analyses were conducted using 45 sets of the evaluation forms.The following sections present the outcomes of the user evaluation.
As shown in Figure 2, most of the respondents are aged below 15 years (10 respondents, 22%), and between 15-20 years (15 respondents, 33%).Further analyses on these two age groups reveal that 12 (27%) of them are secondary school students, and 13 (29%) are secondary school students from the deaf school (Figure 3). Figure 3 also shows that 4 teachers participated in the user evaluation, and one of them is from the deaf school.

User evaluation on e-Sign Dictionary
i.
Use of animated character feature Four different animated characters -man, woman, boy and girl, are used to illustrate the signing of words, as shown in Figure 4 (Ow & Lim, 2008).A user can choose one of the characters to demonstrate the sign for the word.The user has the option to change the character anytime, for example, from a man character to a woman character.The boy and girl characters are provided specifically for young users (Ow & Lim, 2008).As shown in Table 1, evaluation of this feature reveals that 27 (60%) respondents agreed and 17 (38%) respondents totally agreed that the four animated characters (man, woman, boy and girl) are suitable for signing the words.Only 1 (2%) male respondent, in the 15-20 years age group, totally disagreed with the use of the animated characters.This could possibly be due to the fact that he dislikes the four characters which were developed using Poser version 6.0 (Poser 6.0, 2005), and hence, do not have natural human appearance.
ii. Search function One of the main functions of e-Sign Dictionary is the search function.e-Sign Dictionary provides three types of search -by alphabetical order, by category, and by recent search.The categories include fruits, states in Malaysia, colours, vehicles, buildings, occupation, time and date, human body, alphabets, numbers and fractions, and emotions.Recent search allows words that have been searched previously to be repeated.Users do not need to re-type the same words again (Ow & Lim, 2008).As shown in Figure 5, evaluation of the search function reveals that 28 (63%) and 14 (31%) respondents agreed and totally agreed, respectively, that the three types of search function are sufficient.Only 3 (6%) respondents indicated that the three types of search are not enough and other search methods should be provided.The suggestions on other types of search are described in Section 5.
iii.Meaning of words Besides providing three search methods, e-Sign Dictionary also explains the meaning of each word in text and illustrates it with a picture/photo (if available).Feedback shows that 43 (95%) respondents of all races agreed or totally agreed that the meaning of each word is clearly explained.Only 2 (5%) Chinese respondents disagreed, as shown in Figure 6.The respondents were favourable to the use of a picture/photo (if available) to illustrate the meaning of a word.In e-Sign Dictionary, words which can be illustrated using a picture/photo are nouns such as apple and house.All the respondents, irrespective of gender and race, agreed or totally agreed that the pictures/photos provided in e-Sign Dictionary help to illustrate the meaning of the word clearly (Figure 7).

iv. Synonyms and antonyms of words
In a dictionary, it is common to find the synonyms and antonyms of a word.In e-Sign Dictionary, the synonyms and antonyms are usually for adjectives such as beautiful (synonym: pretty; antonym: ugly), and adverbs such as quickly (synonym: rapidly; antonym: slowly).The outcome of the users' evaluation shows that 26 (58%) respondents agreed and 18 (40%) respondents totally agreed that providing the synonyms and antonyms of each word (if available) in e-Sign Dictionary is useful to the learners, as shown in Figure 8. Surprisingly, 1 (2%) Malay respondent disagreed about having this feature in the system.No reason was given to support his disagreement.

v. Contents of e-Sign Dictionary
Currently, a total of 500 words are available in the dictionary.The signing of these words are based on the book entitled "Belajar Bahasa Isyarat Dalam Sepuluh Jam" (meaning: Learn Sign Language in Ten Hours) by Tan Yap, President of the Society of Interpreters of Selangor and Federal Territory, and a Malaysian Sign Language (MSL) expert (Tan, 1998).
Feedback from the evaluation shows that 37 (82%) respondents agreed or totally agreed, that 500 words are sufficient for daily use by the deaf community.However, 8 (18%) respondents disagreed or totally disagreed with such an opinion, as shown in Figure 9.Most respondents suggested that more words should be included in the dictionary.
vi. Instructions displayed on the screen When using any application system, a user often needs to interact with the system to select an option or input a value for processing.This is achieved through simple and easy to understand instructions that are displayed on the screen.As shown in Figure 10, evaluation of the instructions displayed by e-Sign Dictionary shows that 30 (67%) and 13 (28%) respondents, respectively, agreed and totally agreed that the instructions are simple and easy to understand.Only 2 (4%) respondents disagreed.On further investigation, it is found that these two male respondents are Chinese secondary students from the deaf school.It is possible that these two students might not have a good command of English and thus, have difficulty in understanding the instructions of e-Sign Dictionary.

vii. 90° rotation of the animated character
When signing a word, sometimes it is difficult to visualise the hand movement from the front view such as signing the word 'uncle,' as shown in Figure 11.e-Sign Dictionary can illustrate the signing (animation) of a word by rotating 90° to the left, or 90° to the right, as necessary.User evaluation of this 90° rotation feature reveals that 24 (63%) and 18 (31%) respondents, respectively, agreed and totally agreed that this feature helps to illustrate the signing of a word clearly.Three (7%) respondents found such rotation not to be helpful (Figure 12).Thus, this feature needs to be considered when making enhancement to the system.

viii. Zoom-in feature
As the signing of a word may appear to be too small on the screen for some users, e-Sign Dictionary has a zoom-in feature that allows a user to zoom-in on the hand of the human character to view the signing more clearly.The user evaluation of this feature shows that all respondents, except one Chinese respondent (2%), agreed and totally agreed that the zoom-in feature is necessary as it helps in viewing the signing of words clearly (Table 2).On further investigation, it is found that the respondent who disagreed with such feature is a businessman, who views the zoom-in feature as a waste of time because a user needs to zoom-out after using that function.Also, a user will not be able to view the facial expression of the human character especially for adjectives.The facial expressions are also considered a component in the signing of the words.
ix. Ease of use One of the important issues that software developers must consider is the ease of use of a system.If users encounter difficulty when using a software, and there is no help feature, they would be inhibited from using it.Evaluation on the ease of use of e-Sign Dictionary shows that all respondents found it easy to use, except for 3 (7%) female respondents, in the 15-20 years age group, as shown in Table 3.On further investigation, it is found that these 3 respondents are deaf students of Indian ethnicity.They have been using e-Sign Dictionary for between 1-3 months only, and hence, they may need more time to familiarise themselves with it.Further interviews would be conducted with these respondents to understand the difficulties they encountered, in order to improve the ease of use of the dictionary.

Enhancements and improvements to e-Sign Dictionary
This section presents a few recommendations by the respondents to enhance and improve the functions and features of e-Sign Dictionary.

Number of words in the dictionary
Altogether, 17 respondents gave feedback on the number of words that should be stored in the dictionary.The number suggested ranges from 600 -10 000 words, as shown in Table 4.Most (9 out of 17 respondents) suggested having 1 000 words.Unexpectedly, one secondary school student, aged below 15 years, suggested having 10 000 words in the dictionary.Only one deaf student and the deaf school teacher gave suggestions.The deaf student suggested 800 words while the deaf school teacher suggested 1 000 words.According to Tan Yap, a minimum of 800 words that are used daily would be sufficient for communication with the deaf community.

Instructions in other languages
Currently, e-Sign Dictionary provides instructions in English, Malay and Chinese language.Some respondents suggested that including instructions in other languages would encourage other users, who do not understand these three languages but interested to learn MSL, to use e-Sign Dictionary.Table 5 shows the languages recommended by 11 respondents.These languages include: Arabic; Japanese; French; Tamil; and Iban, a language used by the Iban community in Sarawak State, Malaysia.

Additional search method
As discussed in Section 4 above, e-Sign Dictionary provides three types of search.Some respondents suggested that providing a search from sign to word would be useful to the users.Out of the 45 respondents, 16 male and 14 female respondents recommended the proposed search method.To include this type of search into e-Sign Dictionary, all the signs stored in the dictionary need to be classified into two main groups: If this is done, users would be able to retrieve the word in text by choosing the sign being searched from one of the groups.

Translation of sentence from text to sign and vice versa
With the additional search method, it is possible to translate a sentence in text to an MSL sentence, and vice versa.However, feedback from the respondents shows that not all the respondents agreed to have the two-way translation.The opinions of the 45 respondents are shown in Table 6.Out of the 45 respondents, only 21 (47%) respondents suggested having a two-way translation.Thirteen (29%) respondents suggested to have translation from a sentence in text to MSL sentence, while only 3 (7%) respondents suggested the reverse translation from MSL sentence to text sentence.The remaining 8 (17%) respondents did not see the need to have translation at all.The translation from a sentence in text to MSL sentence is easy to implement as the signs can be retrieved from the database of the dictionary, and displayed according to the sequence of the words in the sentence.On the other hand, the translation in the reverse order is difficult, as a sign could have two or more meanings.Hence, the translation of a sentence in MSL to text sentence would require much more efforts to implement as a parsing algorithm needs to be formulated to correctly interpret the meanings of the sign in the context of the constructed sentence.

Appearance of the animated human characters
Although e-Sign Dictionary had received positive feedback on the suitability of using the four animated human characters to illustrate the signing of words, 21 (47%) respondents suggested that the appearance of these animated characters needs to be improved to have a Malaysian look.This would give a Malaysian identity to e-Sign Dictionary as a tool for learning MSL in a multi-racial country.
Besides the five system enhancements and improvements discussed above, 2 respondents from the general public also gave other suggestions to improve e-Sign Dictionary.One of them suggested including speech recognition into the dictionary so that instead of using the keyboard and mouse to select the words being searched, using voice input will make the system easier and more user-friendly, thus, enhancing the usability of the dictionary.Another respondent suggested that an additional window be created, specifically for showing the signing of words using the fingers.

Conclusion
The evaluation of e-Sign Dictionary shows that out of the 45 respondents, 36 (81%) respondents from all races, rated e-Sign Dictionary as a good or very good system.Only 8 (18%) respondents and 1 (2%) respondent rated e-Sign Dictionary as average or poor system, respectively (Figure 13).Further investigation on the respondent who rated e-Sign Dictionary as poor reveals that the respondent is a deaf student of Chinese ethnicity, in the 15-20 years age group.He finds the four animated human characters not suitable for signing words; the three types of search provided by e-Sign Dictionary are not sufficient; and also the meanings of the words are not clearly explained.Hence, he rated the dictionary as poor.
Based on the feedback and comments gathered, out of the ten functions and features reviewed in this paper, nine of them have been rated positively by between 93%-100% of the respondents.Opinions of the respondents on the contents of e-Sign Dictionary that contains 500 words, received 82% positive feedback only.Overall, the outcomes of the user evaluation provide very useful guidelines for future system enhancements and improvements.The feedback, especially from the deaf school teacher and deaf students, are much more accurate and significant than the feedback gathered from the general public.This is because the former groups use the dictionary every day, but the latter groups do not.Nevertheless, all the feedback and suggestions given by the respondents will be incorporated into e-Sign Dictionary.The revised version of e-Sign Dictionary will be updated in the deaf school, and also donated to the other deaf schools throughout the country.The design of e-Sign Dictionary makes it flexible enough to be enhanced to respond to changing user requirements over time so that it will continue to remain a useful tool for the deaf community.

3.
The meaning of each word is clearly explained.

4.
The picture/photo helps to illustrate the meaning of a word clearly.

5.
The synonyms and antonyms are useful to learners.

6.
The 500 words in e-Sign Dictionary is sufficient for communication with the deaf and for daily use.

7.
The instructions on the screen are simple and easy to understand.

8.
The 90 o rotation of the human character to the right or to the left helps to illustrate the signing of words clearly.

9.
The zoom-in feature is necessary as it helps users to view the signing of words clearly.
10. e-Sign Dictionary is easy to use.
11.I would like to use e-Sign Dictionary to learn Malaysian Sign Language (MSL).
12. Overall rating of e-Sign Dictionary.Please tick ( ) one option only.

Part B: Tick ( ) all those items that you think are necessary to enhance the functions and features of e-Sign
Dictionary.

Additional functions and features:
Increase the number of words in the dictionary.Recommended no. of words: _______ Include other languages.Recommended language: ___________________________ Provide search from sign to word.
Provide translation of sentence in text to Malaysian Sign Language sentence.
Provide translation of sentence in Malaysian Sign Language to text sentence.
Improve the appearance of the human characters to have the Malaysian look.This table shows the feedback of the respondents on the use of the four animated characters (man, woman, boy and girl) to sign the words.All respondents agreed or totally agreed with the use of the animated characters, except for 1 (2%) male respondent in the 15-20 years age group, who totally disagreed.table shows the opinions of the respondents on the zoom-in feature of e-Sign Dictionary.All respondents agreed or totally agreed on having the zoom-in feature, except for 1 (2%) respondent who disagreed to having this feature.This table shows the feedback of the respondents on the ease of use of e-Sign Dictionary.All respondents agreed or totally agreed that the system is easy to use, except for 3 (7%) female Indian respondents in the 15-20 years age group, who said that the system is not easy to use.
This table shows the suggestions of the respondents on the number of words that should be in e-Sign Dictionary.
Table 5. Languages recommended by respondents to be included in e-Sign Dictionary Recommended Languages.This table shows other languages suggested by the respondents to be used for the instructions of e-Sign Dictionary.This figure shows the users' evaluation on the use of a picture/photo to illustrate the meaning of a word.All the respondents agreed or totally agreed on the usefulness of this feature.

Arabic
( This figure shows the users' evaluation on providing synonyms and antonyms of words.Only 1 (2%) respondent does not think that this feature is useful.This figure shows the users' evaluation on the 500 words in the dictionary for communication with the deaf and for daily use.A majority agreed or totally agreed that the number of words is sufficient, while 8 (18%) respondents disagreed or totally disagreed.This figure shows the opinions of the respondents on the instructions displayed by e-Sign Dictionary are simple and easy to understand.All respondents agreed or totally agreed, except for 2 (4%) male respondents from the deaf school, who disagreed with this.Totally disagree Disagree Agree Totally Agree (a) hand orientation (i.e. the palm is facing in or facing out), and (b) position of signing (i.e.signing in front or at the left/right side of the forehead, in front or at the left/right side of the face, in front of chest, and at the left/right side of the shoulder).

Figure 3 .
Figure 3. Professions of the Respondents

Figure 6 .
Figure 6.User's Evaluation on the Meaning of the Word

Figure 8 .
Figure 8. User's Evaluation on the Synonyms and Antonyms of Words

Figure 9 .
Figure 9. User's Evaluation on the Contents of e-Sign Dictionary

Figure 10 .
Figure 10.User's Evaluation on the Instructions Displayed on the Screen

Figure 11 .
Figure 11.Signing of the Word 'uncle'

Figure 12 .
Figure 12.Opinions on the 90° Rotation of the Human Character to the Right or to the Left

Table 1 .
Crosstabulation: Age group * Gender * The four human characters: man, woman, boy and girl, are suitable

Table 2 .
Crosstabulation: Race * The zoom-in feature is necessary as it helps users to view the signing of words clearly

Table 3 .
Crosstabulation: Age group * Gender * e-Sign Dictionary is easy to use

Table 4 .
Number of words recommended by respondents to be included in e-Sign Dictionary

Table 6 .
Opinions on the translation of sentence in text to Malaysian Sign Language sentence and vice versa This table shows the opinions of 45 respondents on the translation of sentence in text to Malaysian Sign Language sentence and vice versa.Most respondents (21 respondents, 47%) suggested having two-way translation.Thirteen (29%) respondents suggested having translation from a sentence in text to MSL sentence, while only 3 (7%) respondents suggested having the reverse translation from MSL sentence to text sentence.The remaining 8 (17%) respondents did not see the need for translation.Figure 1.Distribution of Respondents by Gender and Race This figure shows that most of the respondents are Chinese or Malays -15 of them are male Chinese respondents, and 14 are female Malay respondents.
Figure 2. Age Groups of RespondentsThis figure shows the distribution of respondents by age groups.Most of the respondents are of the age groups of below 15 years (22%), and 15-20 years (33%), who are presumably students.