Enhancing Information Systems Auditing Knowledge with Role-playing Game: an Experimental Investigation

This study examined the use and effect of a role-playing game on learners' ability in information systems audit. The study is based on experimental research. Information systems control and audit case study and video had been developed. A total of 75 graduate students undertaking a Master's degree in accounting participated in the experiment. The findings suggested that the role-playing game did enhance participants' perceived knowledgeability, perceived ability to perform actual tasks, and perceived enjoyment of learning. However, learning only occurred if the lecturer was involved in the game. The lecturer had to instruct and provide guidelines to students to perform information systems audit throughout the game. The study also finds an initial relationship between players' enjoyment and their knowledgeability. In addition, the study indicates the correlation between players' enjoyment and their ability to perform the actual information systems auditing tasks.


Introduction
Internal audit survey from Ernst and Young in 2008 indicates the top six areas needing attention: information technologies (IT) (75%), merger and acquisitions (61%), major capital programs (53%), performance improvements (45%), information security (44%), and fraud (39%).The survey also finds that the lack of time for training is the key that hinders auditors' skill development (Ernst and Young, 2008).There are also two main reasons for auditors to understand and evaluate internal control of information systems (IS) used to record and maintain financial information (Curtis et al., 2009).First, many companies are expanding their use of automated information systems.These automated systems may not have suitable control to prevent risks and frauds.Second, some auditing standards, such as SAS No. 109, SOX Section 404, PCAOB's Auditing Standard (AS) No. 5, require auditors to have adequate technical training in information technologies.In addition, future auditors should not only be able to understand and evaluate information systems controls, but also be able to prevent and detect fraud as well as to evaluate the fair value of assets and liabilities.In practice, an auditing firm will separate auditors into generalist or financial statement auditors and IS auditors (sometimes called IT auditors).Usually, generalist auditors may need assistance from IS auditors to evaluate the design and effectiveness of companies' IS control.The current research focused on providing practical training for learners to become IS auditors.
This research was initially developed from the author's enthusiasm for finding teaching methods that effectively equip learners with auditing knowledge and practical skills relevant to today's complex world of information technologies.In addition, most universities conduct their teaching of auditing in a typical classroom lecture style, i.e. a series of 30 lecturing periods (one and a half hour per period).Most students, however, complained that after attending these traditional lecturing sessions, they developed only a shallow understanding of the subject or none at all.They need lecturing sessions that combine an enjoyable practical approach.The classroom lecture needs to be reviewed or improved.In addition, Curtis et al. (2009) call for future research on finding optimal training methods to provide information systems (IS) audit for generalist auditors.The two research questions of Role-playing (RP) is an experience-based instruction.Role-playing activities are often employed within the format of a simulation (Welton and Mallan, 1996;Laney, 2001;Taylor, 2013).Simulation replicates the essential characteristics of the real world but this teaching method does not attempt to generate a perfect real world model.The method simplifies reality so that the real world can be better understood without overwhelming students with too many complex factors (Laney, 2001).Role-playing is normally incorporated in games.All games include some level of competition with the objective of winning (Rieber and Noah, 2008).Normally, role-playing will be adopted in form of role-playing games.
Role-playing in games is a method for students to take the roles of others in order to develop problem solving skills and attitudes (Patil, 2012).The problem solving skills and attitudes are the ability to: (1) identify a problem, (2) generate solutions, (3) evaluate the consequences of solutions to problems, (4) analyse criteria and assumptions behind solutions and (5) acquire new behaviour.Although business schools have used case studies for teaching and learning for a long time, some schools have adopted other teaching methods such as computer games and role-playing games (Brown et al., 2003;Mitri and Cole, 2007;Muncy, 2006;Fand, 2010).In addition, many educators agree that role-playing games are able to support education (Jang and Ryu, 2010).

Processes of IS Audit
IS audit process is normally performed by a group of IS auditors.It generally consists of four stages: planning, preliminary review, fieldwork, and audit report (Alcumus Group, 2013;Montalbano and Contributor, 2013).However, different IS audit firms may conduct the IS audit using a slightly different approach.
The first phase, planning, involves collecting background information related to the audit project.Data may include files of prior audits, policies and procedures manuals, organization charts, job descriptions, sample documents, and other applicable information.This list of documents will be used by IS auditors for planning the audit.During this planning phase, auditors begin drafting an audit program which is basically a list of steps to be performed during the audit.
The second phase, preliminary review, involves reviewing and evaluating the existing internal control structure and making any necessary modifications to the drafted audit scope, objectives, audit program, and questionnaires by IS auditors.
The third phase, fieldwork, typically consists of talking with the company's key personnel, reviewing procedure manuals and reports, learning about the company's business processes, testing for compliance with company policies, procedures, laws, and regulations, and assessing the adequacy of internal controls.These tasks are generally performed on site.Upon completion of fieldwork, the IS auditor summarizes the audit findings, conclusion, and recommendations necessary for the audit report discussion draft.
The final phase, the audit report, is the principal product of auditing.The audit report presents the audit findings and discusses recommendations for improvement.Normally, IS auditors will meet the management team and related personnel to review and discuss the drafted audit report, and agree on any necessary changes to the observations and/or recommendations.However, in some cases, management may choose to either respond to the audit recommendation or not to respond to it and accept the risks associated with an audit finding.The four stages of the IS audit process can be matched with the elements of activity theory which will be discussed in the following section.

Teaching-Methods-Activity-Fit Framework
Various theories from cognitive psychology have been adopted by researchers to explain the effect of teaching methods on user's performance in problem solving.Some studies have applied cognitive theories, such as memory theory and cognitive learning theory (see e.g., Murphy, 1990;Eining and Dorr, 1991).Other studies have used mental models theory (see e.g., Pei and Reaneau, 1990).Finally, some studies have used cognitive-fit theory (see e.g., Vessey and Galletta, 1991;Sinha and Vessey, 1992;Vessey, 1994).However, some previous empirical studies of teaching and learning show mixed results on the benefits of the teaching methods in improving users' performance in learning (Murphy, 1990;Eining and Dorr, 1991;Gregor, 1996).A plausible reason may be the appropriateness of theory used to predict the results.
The researcher believes that the fit between teaching methods and the process of task or activities should enhance knowledge assimilation.The activity theory, however, is more of a descriptive theory than a predictive theory (Nardi, 1995).This theory does not address the effect of independent variables on dependent variables.Meanwhile, cognitive-fit theory posits the effect of fit on performance.Therefore, knowledge assimilation will be enhanced if task and activity are matched.In this research, this matching is called teaching-methods-activity-fit framework (TMAF).
Activity theory is wildly used in education and training.This theory can be used to explain the learning process.In addition, the theory accounts for environment, culture, role of artefacts and complexity of real life activity (Engestrӧm et al., 1999).The theory focuses on practice which will bridge the gap between the individual subject and social reality.Understanding everyday practice in the real world is the objective of teaching IS audit.This theory provides a method of understanding and analysing a phenomenon, finding patterns, and describing phenomena through a built-in language and rhetoric.Activity theory begins with the notion of activity which is a form that a subject does directly to an object in order to obtain a desired outcome.This theory comprises of five related elements: subject, object, community, tools and division of labour.
First, the subject is actors engaged in the activities.The activity theory considers the subject as a team instead of one actor.Among the five teaching methods, a role-playing game also uses two-way communication.Students have to take role to interact with the lecturer, who takes other roles.Second, the object can be a tangible and/or intangible thing.The object, however, should represent reality.Among the teaching methods mentioned, demonstration, case study, and role-playing games involve objects in real life (Jang and Ryu, 2010).Third, the community is a group of actors involved in the activity system.In instructor-led teaching and demonstration, the lecturer is normally the only one who conducts the activities.While the other two methods (learning by teaching and case study), students may have to participate in certain activities depending on the lecturer's teaching strategy.A role-playing game, however, is the teaching method in which students have to conduct activities during teaching processes.Fourth, tools can be anything used by actors in accumulation and transformation of knowledge.All teaching methods have their tools used with students during the class.For example, instructor-led teaching uses PowerPoint presentations while case study uses cases developed from selected companies.Finally, division of labour is hierarchical structure of activities among actors in accumulation and transmission of knowledge.Online game guilds have a hierarchical leadership structure that enables players to complete missions in groups (Chen et al., 2008;Jang and Ryu, 2010).
In addition, Ryu et al. (2005) described three distinct types of learning processes for division of labour: learning-by-investment, learning-by-doing, and learning-from-others. Learning-by-investment is the amount of resources and time that subjects spend to acquire necessary or specialized knowledge for performing the task.Learning-by-doing is defined as accumulation of specialized knowledge by carrying out the assigned task.Learning-from-others is transferred and received knowledge to or from other members within group.Wang and Ramiller (2009) argued that apart from learning-by-doing, learning-about should be addressed for accumulation of knowledge.Learning-about arises from what others have said and written about the specialized knowledge.However, notation of learning-about is similar to learning-by-investment.Therefore, this research examines learning-by-investment instead of learning-about.Of the five teaching methods, a role-playing game involves learning-by-investment, learning-by-doing, and learning-from-others.
The characteristics of the five elements of activity theory are used to derive five factors involved in the matching of teaching methods to activity theory.The factors are subject, object, community, tools, and division of labour.Table 1 presents the teaching-methods-activity-fit framework showing how the fifth characteristics of activity theory relate to teaching methods.This framework indicates that a role-playing game is the most suitable teaching method for teaching IS audit.

Research Model and Hypotheses
Figure 1 illustrates the model of the impact of teaching-methods-activity-fit on learners.The level of teaching-methods-activity-fit affects participants in three aspects: perceived knowledgeability, perceived ability to perform tasks, and perceived enjoyment in learning.Hypotheses developed from this model are discussed as follows.
Figure 1.Model of the impact of teaching-methods-activity-fit on learners De Liso et al. ( 2001) assumed that knowledge increases with high division of labour.Meanwhile, when the level of division of labour, such as learning-by-investment and learning-by-doing, is high, the specialized knowledge of members in a group can be enhanced (Ryu et al., 2005).Because all members of each group communicate during role-playing, they can learn from one another.Therefore, all role playing members should obtain knowledge by communicating with other members (Ryu et al., 2005).Hence, this study proposed that: H1: Students' mean scores of perceived knowledgeability from role-playing game are higher at posttest stage than pretest stage.
A role-playing game can be an effective strategy to use in Science education (Howes and Cruz, 2009).Jackson and Walters (2000) and Smythe and Higgins (2007) reported a great increase in student participation, understanding, and collaboration in a Chemistry course during a role-playing game.Moreover, students will continue to play games if they have an optimal experience doing so (Choi and Kim, 2004).Games may be the manifestation of self-efficacy.Hence, this study proposed that: H2: Students' mean scores of perceived ability to perform the actual tasks from a role-playing game are higher at posttest stage than pretest stage.
Player enjoyment is central to playing games.Enjoyment describes and explains positive and negative reactions derived from games.Fang et al. (2010) stated that enjoyment may be gained in six ways.One of the six ways which can be used to explain the enjoyment of role-playing is the opportunity to explore and experiment with others.In addition, a balance between individual differences in ability to judge game characters and the challenge presented in the game contribute to the enjoyment (Fang et al., 2010).Thus, this study hypothesized that: H3: Students' mean scores of perceived enjoyment of learning from a role-playing game are higher at posttest stage than pretest stage.
Prior research on role-playing games mainly focused on only players' enjoyment.During the game, it is possible that a certain kind of skill might be developed since the subject is exposed to various situations (Jang and Ryu, 2010).Thus, this study hypothesized that: H4a: Perceived enjoyment of learning from a role-playing game is positively correlated with perceived knowledgeability.
H4b: Perceived enjoyment of learning from a role-playing game is positively correlated with perceived ability to perform tasks.

Research Design
A quasi-experimental design with one-group pretest-posttest design was used to gather the data to test the hypotheses.This one-group pretest-posttest design is suitable to test the effect of the training program (Zikmund et al., 2010).The design also offers a comparison of the same individuals before and after training.This design measures the subjects' knowledge of information systems (IS) control and audit before they are exposed to the experimental treatment and then measures knowledge in IS control and audit after they are exposed to the treatment.The experimental treatment in this research is an IS control and audit game.

Construction of the Research Instruments
An IS control and audit case study and video had been built for the role-playing game.The case study in IS control and audit was adapted from Trotman (1995) and Romney and Steinbart (2009).The case study was separated into two parts.Part One was the background information on the company selected as a case study.This includes information about company organization, IT department organization, hardware and application software.This information was provided to students before starting the role-playing game.Part Two was information regarding technology control.This information includes details about physical environment and security, logical security, operation procedures, system standards, logging, disaster recovery planning and application audit (sales and account receivable, purchase and account payable, and production).This second part of information allows the instructor to assume the roles of several actors in the organization chart (e.g. company director, IS manager, security guard, production manager, sales manager, and accounting manager) (Figure 2).The instructor used the second part of information to answer questions asked by students who played the role of IS auditors.A video was provided to describe the physical existence of computer centre of company being audited.The instructor would play this video when students wanted to visit the computer centre.Figure 3 shows one shot screen for computer centre from video.
In the questionnaire, participants gave their answers by rating a five-point Likert scale.The scale ranges from 1 for "strongly disagree" to 5 for "strongly agree".A higher score indicated stronger agreement.A five point Likert scale was also used to rate the perceived knowledgeability, perceived ability to perform actual tasks, and perceived enjoyment in learning.
Questions in the questionnaire for the first two variables in the research model (Figure 1) were originally developed by the author while the last variable was based on Fang et al. (2010) and Chen et al. (2012).In order to measure perceived knowledgeability, two items were included.These items are "I have gained knowledge necessary for performing IS controls through the role-playing game" and "I have gained knowledge necessary for performing IS audit through the role-playing game".In order to measure the perceived ability to perform the actual tasks, two categories of items were included.These items are "I could imagine how an IS auditor works in real life from the role-playing game" and "I could learn the actual IS auditing tasks from the role-playing game".With regard to enjoyment in learning, three items were included.These items are "I forgot about the passing time while playing the role-playing game", "I felt emotionally involved in the role-playing game", and "I want to learn other areas of IS audit through the role-playing game".Though the sample size is not sufficient to perform factor analysis to assess construct validity, the reliability of the questionnaire has been analysed.The value of Cronbach's alphas for pretest of perceived knowledgeability, perceived ability to perform actual tasks, and perceived enjoyment in learning are 0.824, 0.670, and 0.733 respectively.The results of Cronbach's alphas for posttest of perceived knowledgeability, perceived ability to perform actual tasks, and perceived enjoyment in learning are 0.754, 0.728, and 0.668 respectively.Though some factors do not show high value of Cronbach's alpha, the result is greater than 0.60.Moore and Benbasat (1991) suggested that in early stages of research, reliabilities of 0.50 to 0.60 would suffice.Therefore, the questions in the questionnaire are statistically reliable.
To avoid bias and ambiguity in the research instruments, a pilot test was conducted by a senior IS auditor in a state university in Thailand.The auditor has worked in this area for almost 15 years.This auditor took one hour test.After the test, the research instruments (IS auditing case study, video of computer centre, and questionnaire) and experimental procedures were adjusted based on the auditor's comments.

Participants
This study was conducted by the researcher during the course of an information systems control and audit subject given at two large public universities in Thailand.The participants were 75 graduate students undertaking a Master's degree in accounting.The reason for limiting the study to these accounting students was that participants in this area have better fundamental knowledge regarding financial statement audit than participants from other areas.Apart from IS knowledge, this financial auditing knowledge is the fundamental principle needed to work as IS auditors.In addition, some participants worked as external or internal auditors.The researcher administered a pretest on all participants.After the treatment the researcher administered a posttest on all students and compared the results.The data was analysed and interpreted to test the effectiveness of the roleplaying game teaching method.

Experimental Procedures
The experiment was conducted in the lecture rooms of each university.Students were divided into five IS audit groups (two groups for auditing data centre and three groups for auditing application which had been subdivided into revenue cycle, expenditure cycle and production cycle).The experiment ran over six sessions (three hours per session) and consisted of four phases.
Phase I involved briefing the group by introducing the general story of an IS audit case and explaining the role play to each group.This phase took one session.
Phase II involved planning the audit and designing an IS audit questionnaire.Each group brainstormed and prepared an IS audit plan and questionnaire for their audit tasks.This phase took one session.
Phase III was enacted by beginning the role play.Each group assumed the role of IS auditors who analysed data collected from the questionnaire.Students, however, could explore the computer room or ask for related IS documents such as security policy and contract of hot site for disaster recovery plan.Meanwhile, the lecturer assumed the roles of the company staff.The lecturer provided answers to students' questions and helped students explore the art of interviewing (asking the right question to the right person).The lecturer also provided support and guidance for students on how to evaluate answers that are valuable for IS auditing and encouraged students to express ideas and feeling.This phase took three sessions.
Phase IV was the discussion and preparation of the IS audit report.Students collected all information for the case by interviewing staff, surveying the computer room (data centre), employing CAATs (computer-assisted audit techniques) software packages such as ACL and IDEA with provided company files, and inspecting relevant documents obtained during phase III.They eventually prepared the IS audit report and discussed the weaknesses of IS within the case study.During the discussion of audit finding, the lecturer acted as the company staff, discussing and defending against the audit finding with students in each group.After finishing the discussion, students concluded the IS audit report.This phase took one session.

Results
This section reports the results of the assessed variables relating to the assumptions underlying the statistical tests and the research findings.A statistical package was used to evaluate the assumptions before proceeding with a paired-sample t test and simple regression.

Evaluation of Assumptions
Seventy five graduate students had completed the experimental procedures and their responses were included in the analysis.The participants comprised of 21.33 percent males and 78.67 percent females.There were 52.0 percent of participants aged 21-26 years old; 40.0 percent of participants aged 27-32 years old; 4.0 percent of participants aged 33-38 years old; and 4.0 percent of participants aged over 39.Among the sample, 10.7 percent of participants were students, 56.0 percent were government and state enterprise officers, 20.0 percent were private company officers, and 13.3 percent were in the 'other' category.Results for the evaluation of assumptions of normality, homogeneity of variance, linearity, and multicollinearity of each dependent variable were satisfactory.

Analysis of the Results and Discussion
A paired-samples t-test was used to analyse the effect of teaching-methods-activity-fit for H1 to H3 on the data of 75 participants.The analysis was performed on dependent variables (perceived knowledgeability, perceived ability to perform actual tasks, and perceived enjoyment in learning).The independent variable is level of teaching-methods-activity-fit.Meanwhile, a simple regression was used to analyse the effect of perceived enjoyment in learning on perceived knowledgeability and perceived ability to perform tasks of H4a and H4b respectively.A p-value of 0.05 was used as the significance level in the statistical tests.A p-value between 0.05 and 0.10 is deemed as marginally significance.
H1 suggested that students' mean scores of perceived knowledgeability from role-playing game were higher at the posttest stage than at the pretest stage.The results of a paired-samples t-test in Table 2 indicated significance for the effect of level of teaching-methods-activity-fit on the perceived knowledgeability (t=20.835,p < 0.001, see Table 2) Meanwhile, mean value of perceived knowledgeability at the pretest stage was 2.146.The mean value of perceived knowledgeability at the posttest stage was 3.733.The level of teaching-methods-activity-fit appears to affect the perceived knowledgeability.Thus, H1 is supported.This result is inconsistent with the study done by Rieber and Noah (2008) who found that games interfered with participants' learning of scientific principles.One of the reasons may be the involvement of the lecturer as an actor in role-playing game.The lecturer instructed and provided guidelines to students in performing the IS audit throughout the game.Thus, the learning effect is mediated by the lecturer.
H2 suggested that students' mean scores of perceived ability to perform actual tasks from role-playing game were higher at the posttest stage than at the pretest stage.Statistical significant support was obtained for the effects of level of teaching-methods-activity-fit on perceived ability to perform actual tasks (t=16.485,p < 0.001, see table 2).Meanwhile, the mean value of perceived ability to perform actual tasks at the pretest stage was 2.646.The mean value of perceived ability to perform actual tasks at the posttest stage was 4.040.The level of teaching-methods-activity-fit appears to affect the perceived ability to perform actual tasks.Thus, H2 is supported.
H3 suggested that students' mean scores of perceived enjoyment of learning from role-playing game were higher at the posttest stage than at the pretest stage.Statistically significant support was obtained for the effects of level of teaching-methods-activity-fit on perceived enjoyment in learning with a t-value of 12.426, p <0.001 (see table 2).Meanwhile, the mean value of perceived enjoyment in learning at the pretest stage is 2.840.The mean value of perceived enjoyment in learning at the posttest stage was 3.973.The level of teaching-methods-activity-fit appeared to affect perceived enjoyment in learning.Thus, H3 is supported.This result is consistent with that obtained by Chen et al. (2012) who found the influence of game quests on pupils' enjoyment in math learning.H4a suggested that students' perceived enjoyment of learning from role-playing game is positively correlated with students' perceived knowledgeability.Statistical significant support was obtained for the effects of perceived enjoyment of learning on perceived knowledgeability with a p-value of 0.007 (see Table 3).Detailed information of t test on perceived enjoyment showed statistical significance with t = 2.774, p = 0.007.In addition, the relationship between perceived enjoyment and perceived knowlegeability was positive.Therefore, the results provided strong support for H4a.H4b suggests that students' perceived enjoyment of learning from role-playing game is positively correlated with students' perceived ability to perform tasks.Statistical significant support was obtained for the effects of perceived enjoyment of learning on perceived ability to perform tasks with a p-value of 0.000 (see Table 4).
Detailed information of t test on perceived enjoyment showed statistical significance with t = 4.157, p = 0.000.
In addition, the relationship between perceived enjoyment and perceived ability to perform tasks was positive.Therefore, the results provided strong support for H4b.

Discussion
This study shared a role-playing game experience in an IS audit according to the model of the impact of teaching-methods-activity-fit on learners.This study was conducted as an experiment in order to find the answers for the research questions -"What are the optimal teaching methods for informing learners about the IS auditing process?" and "What are the influences of the role-playing game on participants' learning process in terms of perceived knowledgeability, perceived ability to perform tasks, and perceived enjoyment of learning?" The current study is expected to contribute to the literature in two ways.First, the findings reveal that teaching-methods-activity-fit affects participants in three levels: the perceived knowledgeability, the perceived ability to perform actual tasks, and the perceived enjoyment in learning.The study also finds an initial relationship between players' enjoyment and players' knowledgeability.In addition, the study indicates the correlation between players' enjoyment and players' ability to perform the actual information systems auditing tasks.Second, this study raises several issues regarding the design of role-playing games.The study supports the notion that role-playing game can be used as an effective teaching technique since learners can achieve their mission in the experiment with step by step procedure.The procedure of this experiment allows students to interact with other players and the lecturer, who takes specific roles.As one of the actors, the lecturer can monitor and promote student understanding and learning (Rieber and Noah, 2008).
The study has some limitations that are also opportunities for future research.First, the results of the experiment may have been affected by internal validity.The time lapse between pretest and posttest was a period of several weeks, so students may have matured as a result of experience on the subject (maturity effect).This effect may also have influenced the dependent measure in this design.The effect of testing may also have confounded the experiment.However, while there might be some questions about internal validity, the researcher must always take into account the notion of time and cost in most business research (Zikmund et al., 2010).Future studies may pursue this issue by conducting other research with a design to control internal validity.
Second, little theoretical work exists regarding use of role-playing games.Therefore, future studies could be conducted in areas related to the impact of the matching of role-playing games and players' optimal mentality on player's capability.In addition, the studies may also posit a possible relationship between players' enjoyment and their knowledgeability and the correlation between players' enjoyment and their ability to perform actual IS auditing tasks.The initial statistics in this study support these relationships.The researcher suggests a further need for a qualitative research since not much has been conducted in these areas so far.The qualitative method may reveal and clarify the influence of players' enjoyment on students' learning called from Rieber and Noah (2008) and Chen et al. (2012).
Finally, this research measures enjoyment only on affective dimension which focuses largely on empathy and positive moods.As measurement of enjoyment incorporates of three dimensions (affective, cognitive, and behavioural dimensions) (Fang et al., 2010), future research may attempt to measure enjoyment within three dimensions for an in depth view on IS auditing learning ability.

Figure 2 .
Figure 2. Actors in the role-playing game

Table 2 .
Summary table of t-test for questionnaire

Table 3 .
Effect of perceived enjoyment of learning on perceived knowledgeability

Table 4 .
Effect of perceived enjoyment of learning on perceived ability to perform tasks