Online Quizzes for Operations Research – A Case Study

A student's success in mathematically-based disciplines is directly related to the quantity and quality of the tasks provided, and to the feedback given on their efforts. With a class of several hundred students it is often impossible to provide enough assessable work, and to give detailed and timely feedback. In response, the Department of Statistics at Macquarie University has implemented online randomised quizzes. Students must pass each quiz with at most two errors, but the number of attempts is unlimited. Consequently, an assessment task becomes a learning tool, requiring students to practice techniques until mastered. Feedback is immediate.


Background
Economics, accounting and statistics students in their third year at Macquarie University undertake the unit Operations Research II (STAT379), which has as a prerequisite, Operations Research I (STAT279), a second year unit.On average 500 students enrol in STAT279 per semester, while 150 students elect to do STAT379, on average, per semester.For several years, the coursework component of both units consisted of a class test, weekly homework exercises (which were marked on attempt only), and for STAT279, two paper-based multiple choice quizzes.There was a heavy weighting towards the final examination.In both units the failure rates had typically been higher than desired.Because of the large enrolments it was difficult to provide more assessable work and useful feedback to support the students' learning.Staff felt that these difficulties were affecting both students' engagement with the material and their approach to study.This is a common problem faced by university mathematics educators, and Groen (2006) and Keady, Fitz-Gerald, Gamble and Sangwin (2006) are among those who have developed computer-aided assessment as a solution.Groen found that students showed no preference for online assessment over traditional methods and that open-access conditions led to improved marks.For the staff member there were also efficiency benefits for marking, administration and resource use.These authors point out those economies of scale can mean that the more students who use a computer-aided system, the more worthwhile it becomes.Keady et al. recommend that the management goal for these systems should be keeping them simple, and note that increasing popularity of these systems will lead to continuing improvement.Jacobs (2005) proposes that visual presentation can assist with learning difficult mathematical concepts, which computers and web-based applications can provide particularly when allied with interactive tools and animation.This paper deals with the development of an online differential equations course with graphics and interaction as key features of the course.Online assessment tasks were also developed, in the form of, "randomized multiple choice quizzes and randomized worksheets" (p.765),where students were prompted to repeat question types until they can answer correctly.In a further paper, Jacobs (2006) states that the computer and web tools available can, "enhance enjoyment and satisfaction as well as presenting a broad visual overview to improve conceptual understanding" (p.1).Online assessment can be combined with the teaching mode to benefit the student's understanding.Online modules were created for two courses, "using a student-centred, interactive approach that adheres to the fundamental constructivist guidelines …" (p.2).
The success of blended learning methods (online and face-to-face) was investigated by Kaczynski, Wood and Harding (2007, in press) using a radar chart method of analysis, based on self-assessment of learning alongside a range of qualitative research methods.The authors found that online delivery should be used with caution since, "excessive online content appears to more rapidly overwhelm the student."(p.14).

Online quizzes used for learning, not assessment
In the past, paper-based quizzes were used in STAT279 to assess students' progress during semester, but STAT379 had no quizzes at all.The paper-based quizzes were used simply as an assessment tool, and as a way to ensure that students mastered some material part way through the semester.There was a considerable lag time (two to three weeks) between submission and return of the marked assessment task.Therefore, the paper-based quizzes were of no use in identifying struggling students early in the semester.Moreover, any feedback on the students' performance or on students' ability to cope with the subject matter generally came too late, making it impossible for them to address and rectify problematic areas.Furthermore, the paper-based quizzes created a heavy marking and administrative workload for academic staff.These difficulties prompted the search for an alternative solution, and so the possibility of using an online quiz system was explored.
The requirements for the type of quiz system were twofold: that students would gain proficiency and understanding of the topics presented, and that staff would be freed from large amounts of marking and administration.The former requirement led to a preference for a system that functioned as a tool for formative, rather than summative, assessment.Such a system would allow students multiple attempts at a quiz, provide feedback on their responses, give detailed solutions, and have a variety of question types other than simple multiple choice.The system chosen, MacQTeX, satisfied all these requirements and also included sophisticated randomisation, a text-entry question type, as well as two variants of multiple choices.Furthermore, MacQTeX is produced within the same university, facilitating customisation of the system and liaison with the developers.Details of the system will be given in Section 2.
During Semester 1, 2006, the paper-based quizzes in STAT279 were replaced by online quizzes, which were designed as a learning tool for students rather than simply a way to assess their progress.In order to emphasise learning rather than assessment, the students were expected to work on the quizzes until mastered.To make sure students participated fully a token weighting was given to the quizzes, based on a pass but not on the actual mark.Students were allowed to attempt each quiz as many times as they needed in order to pass it.A quiz is considered passed if the student makes at most two errors, where the quiz has between 12 and 20 questions or question parts.The randomisation feature of the online quiz system ensured that students were always presented with a different version of the quiz.After submitting their answers to a quiz, students were instantly provided with fully-worked solutions to the problems and feedback on their performance.This instant feedback enabled students to study the solutions and methods prior to making another attempt at the quiz.With the online quizzes, students' progress and workload were more evenly paced throughout the semester than had previously been the case.Furthermore, the online quizzes were shown to boost students' problem-solving skills and understanding of the subject matter noticeably (Griffin & Gudlaugsdottir, 2006).
Following the successful implementation of the online quizzes in STAT279, it was decided to design a similar set of quizzes for STAT379.Students progressing to STAT379 had indicated to lecturing staff that they had found the STAT279 quizzes in the previous semester very useful, both as an aid to their understanding of the concepts and as a means of consolidating their knowledge and techniques.A small set of online quizzes for STAT379 was trialled during Semester 1, 2007, and a larger improved set was implemented the following semester.
The design, writing and testing of the quizzes for STAT379 was funded by the Division of Economic and Financial Studies, which offers the unit.The development of new questions involved identifying the types of routine problems that would benefit the students, leading to increased technical facility and sound understanding of the underlying statistical theory.Appropriate scenarios which allowed sufficient variety and scope for randomisation had to be designed, and the quiz system itself was extended to allow for several methods of scoring answers containing multiple text-entry fields.
During the last week of Semester 2, 2007, students enrolled in both STAT279 and STAT379 were invited to participate in a survey.This was designed to provide information on students' reactions to the quizzes as an aid to learning, and to assess the usability of the system.Participation in the survey was entirely voluntary and students' responses were anonymous.In total, 375 STAT279 and 84 STAT379 students participated in the survey.This gave participation rates of 76% and 53% for STAT279 and STAT379 respectively.Results from this survey will be presented and discussed in Section 4 of this article, as well as being referred to in Section 2.
The outline of the article is as follows: in Section 2, a description of the MacQTeX online quiz system is given, including a brief outline of certain technical aspects of the system.The implementation of the online quizzes and the design of the student survey is outlined in Section 3. The results from the survey are presented in Section 4 and, following on from that, the effect of the online quizzes on teaching and learning is discussed in Section 5. Section 6 contains concluding remarks.

Quizzes
The MacQTeX online quiz system was developed in the Department of Mathematics, Macquarie University, and has been used for several years by that Department.The quizzes do not require students to install any special software, and are accessible at any time and from any location.The correct and incorrect answer choices are indicated, and a hint is displayed to explain why the selected answer choice is wrong.
The quizzes are presented as interactive PDF documents, containing form fields similar to those one would see on a standard web page.The question types include multiple choice, multiple response (the correct answer is comprised of several choices) and fill-in-the-answer.The last accepts numeric, text or mathematical expressions as input, with the ability to interpret mathematical expressions and recognise those that are equivalent.Multiple response questions are particularly suited to helping students remember definitions, or to interpret output from data analysis software.Examples of a fill-in and a multiple response question are shown in Figures 1 and 2 respectively.
All processing of answers and calculation of scores is self-contained.This provides much faster operation than in many other quiz systems, which contact a server for every action the student takes.A further advantage of client-side processing is that a student may review and change answers at any time until the completed quiz is submitted to the server.Of students surveyed, 80% (STAT279) and 86% (STAT379) responded that they liked this feature.
Various parameters in the questions are randomised, so that students may attempt many versions of the same quiz, thus gaining valuable practice.The randomisation can apply not just to numeric parameters, but also to the form of mathematical expressions, such as the type of function or dimensions of a matrix, and to scenarios relating to data sets.On completion of a quiz, it is marked automatically and fully-worked solutions become available, giving students immediate feedback.The solutions are not simply models, as all calculations are performed with the specific random parameters used in the questions.An example involving graphics can be seen in Figure 3.According to the student survey, the level of detail in the solutions was considered sufficient by 74% (STAT279) and 70% (STAT379) of participants.Figure 3.An example of a fully worked solution.
The graphs are generated using the specific inequalities shown in the question as constraints.
For any multiple choice or multiple response question that the student has answered incorrectly, hints become available indicating what was wrong with the particular answers chosen.In many cases these hints are designed to prompt the student to think more deeply about the problem or to recall certain facts, rather than simply pointing out the specific error that may have been made.A hint is shown in Figure 2, and appears when the cursor is passed over the incorrect answer choice.
Questions with several parts, all of which use the same random parameters or data set, can also be generated.Since the answers to such multi-part questions are interdependent, a student may choose to check his/her answer to an early part in order to be sure of using correct information for subsequent parts.Once the preceding answers have been revealed, they may no longer be changed.This feature was considered useful by 80% of survey participants in both units.

Quiz management tools
An extensive set of quiz and class management tools is available to staff through a simple web interface, as is the setting up of new quizzes.To create a new quiz, a lecturer simply chooses the questions from a catalogue of existing questions and decides on the colour scheme and level of server interaction.Quizzes come in three flavours: the first records all students' activity, including scores, on the server; the second records only quiz downloads but not scores; the third functions without server interaction, and is intended for use on a CD where JavaScript in a web page would randomly choose which version of the quiz to present.
After the questions have been chosen, a sample quiz is generated and presented for checking.Once satisfied, the lecturer then triggers the generation of the first set of randomised quizzes.When these are almost used up by the students, another set is automatically generated.Quizzes are encrypted, ensuring that the answers and solutions are inaccessible in Acrobat Professional.
Expiry dates for the quizzes can be set if desired, and can be changed at any time during the semester.Pass marks for quizzes can be chosen, otherwise the default is full marks for a pass.Generally one or two errors are allowed to ensure that students who may mis-type an answer to a fill-in question still have a chance of passing the quiz without undue frustration.Students who miss the deadlines can have their access reinstated for a certain number of days.A student who has not passed a particular quiz can be granted a pass at the discretion of the lecturer, for example in the case of illness or misadventure.
Class statistics are available in both numeric and graphic form.Numeric displays include numbers of downloads, submissions and successful attempts.These counts represent both total activity and numbers of individual students who are active.Average and maximum numbers of attempts are also included.Graphic displays of usage patterns include overall usage, daily and hourly averages, overall activity for each quiz and numbers of incorrect responses for each question.These can be seen in Figures 4 and 5.Each student can view his/her personal quiz log as a web page.This shows pass status and best score for each quiz; access details; scores and answer choices for every attempt at each quiz; times taken for perfect scores; and records of access being denied or reinstated.Class summary statistics can also be dynamically displayed on the unit website.

Quiz production and delivery
The MacQTeX quiz system runs on a Mac OS X server, but could be installed on any other UNIX platform.It uses free software only, including PERL, mySQL and the mathematics typesetting engine pdfLaTeX.Class lists, student records, quiz activity and various parameters such as pass marks, expiry dates and question lists are stored in mySQL databases.The quiz delivery, quiz management and randomisation are all controlled using PERL.The random parameters are programmatically generated on the fly for each quiz, not selected from a database, which differentiates it from many other quiz systems.
PDF was chosen as the format for presentation because it allows the inclusion of JavaScript controlled form fields, similar to those common on standard web pages.The presentation of mathematics on web pages is generally unsatisfactory, whereas with PDF there is complete control over the layout and appearance.There is the added advantage that pdfLaTeX can be scripted to generate documents containing complex JavaScript functionality.

Quiz question design
The amount of variety in the random parameters of a question is best determined by educational considerations rather than by the limitations of programming practicality.The former, in most cases, often leads to a simplification in the latter anyway.The decision as to whether a question should be fill-in or multiple choice will be guided by the complexity of the mathematical expressions needed for text entry, the possibility of writing appropriate distractors for multiple choice, and whether a multiple choice form of the question would make the correct answer too obvious.On the other hand, multiple response questions are very useful for checking recall of definitions, interpretation of graphs and output from statistical packages.
Certain cases of a particular question may present greater difficulty to students than others.Hence the randomisation can be designed so that these cases occur more often, thus providing students with extra practice.A problem in which a simple case is taught first, and more complicated cases later, is better programmed as several different questions.In this way students are able to practise and gain confidence with the straightforward case before tackling the more difficult cases.
In developing randomised questions, the random parameters must be chosen so that the tedium of any numerical calculations does not overwhelm the concept that the question is intended to reinforce.On the other hand, unfortunate choice of parameters can sometimes lead to a trivial case of the question so this must be avoided.The complexity of mathematical expressions to be entered as answers should also be kept to a minimum.
Once generated, mathematical expressions must be filtered so that fractions and surds are reduced to lowest terms; instances of silly notation such as 1x or x 0 are removed; and an appropriate level of factorisation is carried out.This is important as students should be encouraged to present their solutions using succinct and precise mathematical language.In text that describes the problems, the grammar must be adjusted to account for plurals and so on.In addition, the tolerance of precision required for fill-in answers can be adjusted to allow students some lenience in the manner of rounding their decimals.The use of decimals and certain other forms of mathematical notation can even be prohibited in particular questions, if desired.(For more detailed descriptions for any of the aspects outlined above see Griffin, 2005;Griffin, 2004;Griffin & Moore, 2003.)

Use of online quizzes
During Semester 2, 2007, 493 and 160 students were enrolled into STAT279 and STAT379 respectively.The online quizzes formed a compulsory part of the semester coursework, contributing a small percentage of the marks in their final grades for the unit.For STAT279 there were four quizzes, with three for STAT379.Each quiz contained four or five multi-part problems.For both units, the first quiz was on assumed knowledge with the remaining quizzes being on new material covered during the semester.Students were able to do all the quizzes as many times as they wished and they were considered to have passed a quiz if they had two errors or less in a quiz with from 12 to 20 questions or question parts.Tables 1 and 2 show the participation and pass rates for the online quizzes for STAT279 and STAT379.As expected, these rates drop off towards the end of semester as students become too busy with work in other units.In addition, the decrease represents the presence of students in the units who are not really serious about performing well, some of whom drop out of the unit but do not formally withdraw their enrolment.Most of these non-participating students do not sit the final exam.

Design and administration of the student survey
In order to gain a clear understanding of student's perception of the online quizzes as a learning tool, students from both units were invited to participate in a survey.The survey was designed to provide information on: (1) the extent to which students felt that the quizzes enhanced their learning; and (2) students' perception of how easy the MacQTeX online quiz system was to use.
The survey contained 23 statements and participants were asked to indicate their level of agreement to each of the statements on a five-point Likert scale: Strongly agree, Agree, Neutral, Disagree, Strongly disagree.The option of Not Applicable was given as well.In order to simplify the results, they are presented below with the Strongly agree and Agree categories combined into one "Agree" category, as the proportion of responses in these categories was consistently very much higher than in the other three categories.Similarly, the Disagree and Strongly disagree categories have also been combined into one "Disagree" category.For all statements a very small percentage responded with Strongly disagree, so combining this with the Disagree category is not misleading.
The survey was administered in the tutorial classes for both units during the last week of the semester.Participation in the survey was voluntary and the responses were anonymous.It was administered by a person who was not directly involved in the design of the online quiz system or this research.The students were given approximately 10 minutes to complete the survey.

Survey results
As previously mentioned, altogether 375 STAT279 and 84 STAT379 students participated in the survey, giving participation rates of 76% and 53% for STAT279 and STAT379 respectively.In general there was strong agreement in the responses to the survey statements between students from the two units.
Figure 6.Effect of online quizzes on students' perception of learning and understanding of unit content.
The survey contained three statements that were designed to assess the students' view on how the quizzes assisted them with the learning and understanding of the subject matter.
Statement 1: The quizzes helped me understand the concepts in this unit.
Statement 2: The quizzes helped me learn how to express my solutions in a clear and logical way.
Statement 3: Doing the quizzes has helped me to remember the processes for solving particular problems.
Figure 6 shows that more than 70% of participants agreed that the quizzes helped them to understand the concepts in the unit; to learn how to express their solution in a clear and logical way; as well as to remember the processes for solving particular problems.The percentage of participants who agreed was slightly higher for STAT379 than for STAT279 for statements 1 and 2, and 10% higher for statement 3.This may indicate that students in their third year have a more mature and structured approach to study.
Of the survey participants, 81% percent of STAT279 and 88% of STAT379 liked the way the quizzes provided them with quick feedback on their progress.Furthermore, 61% of STAT279 and 70% of STAT379 survey participants thought that the benefits they gained from the quizzes were worth the extra workload they created.
There were five statements that assessed students' views on their study habits in relation to the online quizzes.These statements were included in order to gain insight into the way students behave in relation to study planning, time management, deadlines and exam preparation.The responses are summarised in Table 3.The results were somewhat surprising, as 63% (STAT279) and 69% (STAT379) of survey participants indicated that they began work on the quizzes well before the due date.This is contrary to the impression gained by staff that students habitually start work late on their assessment tasks.It is also not entirely borne out by the quiz logs, which show that quiz activity on the due date and the previous day is much higher than at any other time, as shown in Figure 4.There is the possibility that the sample of students attending classes in the week of the survey represented those who were generally more organised and conscientious, or that students' idea of "early" is not the same as that of staff.The lower percentage (46%) of STAT379 than STAT279 (51%) survey participants using the quizzes for final exam revision is most likely due to the timing of the exams in the two units.The STAT279 final exam was scheduled in the week following the survey, whereas the STAT379 exam was two weeks later, so STAT379 students are unlikely to have started exam revision at the time of the survey.Again, the quiz logs do not really support this, with only a small amount of quiz activity leading up to the exams.It could be that at the time of the survey, many students had the intention of using the quizzes for revision but eventually did not.
The low percentage of survey participants indicating that they collaborated with other students on the quizzes is also surprising.This was unexpected since the cohorts contain a large proportion of students whose first language is not English, and it is known from observation and conversation with these students that one of their study strategies is to work together, particularly where interpretation of a scenario described in words is involved.
In response to the statement, "It was helpful to be able to do the quizzes as many times as was needed to pass", 79% of all the survey participants agreed.
As there was a large number of international students having English as their second language undertaking these two units, there has been some concern that the wording of the problems presented in the quizzes may have been too complex.The statement in the survey that assessed students' views on this matter revealed that 71% of STAT279 and 83% of STAT379 survey participants thought the questions presented in the quizzes were worded in a way that was easy to understand.Furthermore, for both units overall, 81% of survey participants thought that the difficulty level of the problems was "just right".

Implications for learning and teaching
Online quizzes, such as those used in STAT279 and STAT379, are intended to enable students to gain proficiency at solving routine problems.Without such proficiency it would be much harder for them to learn to perform the more complicated techniques and algorithms required in the statistics discipline.Having provision for automated routine exercises allows for the use of more interesting problems in written assessment tasks, which now need not contain the routine exercises.
Both units require students to use and be proficient with statistical packages such as QuickQuant.They must know how to set up problems and give detailed interpretations of the output.With the inclusion of the online quizzes, students are also required to demonstrate their ability to perform similar calculations by hand, thus encouraging deeper understanding of the processes involved.In addition, the quizzes include synthetic QuickQuant output, which the students must then correctly interpret.
The survey results show that the unlimited number of attempts students can have at solving the quizzes, the instantaneous feedback on their performance, as well as the full solution that is provided after submission has taken place, encourage self learning.It has been observed that students are more likely to attempt a quiz again within a short period of time if a detailed solution is provided.Furthermore, students' scores have been observed to improve rapidly from one attempt to the next.When asked, both in the survey and informally, students said they learned a lot about analytical skills, problem-solving techniques and the subject matter generally from looking through the fully-worked solutions.
It has also been found that weaker students typically repeat a quiz many times before attaining the required passing level.Therefore the unlimited number of attempts provides these students with valuable practice they would otherwise not have been able to obtain.Students often submit a quiz without attempting any questions before making a real attempt at the quiz, in order to study the solutions so that they can learn the material.
The assumed knowledge quizzes for both STAT279 and STAT379 were designed to give an indication of whether students have the required background knowledge and technical skill to provide sufficient foundation for the unit subject matter.By requesting students to attempt these quizzes within the first three weeks of the semester, lecturing staff are able to identify struggling students early on and assist them with catching up on the background material.These students are then invited individually to discuss their difficulties with the lecturer, who may then provide them with further study materials according to their needs and follow up on their progress later in the semester.
The online quiz system can also provide lecturing staff with an immediate indication of the level of understanding the students have reached on the subject matter taught during the semester.Written assessments, such as assignments or class tests, have been used commonly to identify topics that are causing difficulties for students.Lecturers running units with large enrolments may find this impractical.Large units often involve many markers where variation in marking, as well as a lack of overview, can make it very difficult to identify such topics accurately.The time taken to manage the distribution of marking and the return of work to students makes any feedback less effective, as students' memory of the processes they used to solve the problems has faded.
This time lag also makes it difficult to obtain any reliable statistics on student performance.The online quiz system used for STAT279 and STAT379 provides graphs (such as the one presented in Figure 5) that enable the lecturer to see immediately whether an individual topic is problematic for students.After ensuring that it is not the question itself or the programming that is at fault, actions can be taken in order to clarify the subject matter for the students.This clarification can be implemented during lectures, tutorials or by using electronic bulletin boards.In order to assess whether the lecturers' intervention was successful, follow-up questions on the same topic can be included in a quiz later on in the semester.
Academic staff involved with the marking of other assessments tasks (such as class tests and final exams) for both STAT279 and STAT379 have commented on a noticeable improvement in students' attempts at solving routine problems, as well as in the way in which they tackle the more involved and complex problems.It appears that students have gained a better appreciation of how to break a problem down into a sequence of tasks and how to express their solution in a structured and coherent way.Furthermore, there has been a noticeable improvement in the pass rates in both units; this cannot be entirely attributed to the online quizzes, since there have been other changes to the unit assessment requirements and delivery of lecture and tutorial material.

Conclusion
The introduction of online quizzes into both STAT279 and STAT379 was highly successful.The verbal feedback from students was positive, and their wishes on entering STAT379 to have quizzes similar to those used in STAT279 indicated that students found the quizzes useful.This impression was clearly supported by the survey.
The effect of identifying struggling students early in semester is difficult to quantify, but it most certainly made a difference to the experience of certain individuals.Without the prompt intervention of teaching staff these students may well have failed, or simply dropped out of the unit.These are students who either found the material very challenging, whose motivation was poor, or who have had difficulties in their personal lives affecting their study.The individual attention given by staff was most appreciated, and assisted them in overcoming their problems and succeeding in the units.
It was fortunate that an existing quiz system was available so that it was possible to introduce it over a short timeframe, without the worry of technical problems or significant usability issues.STAT279 was the largest unit which has used the quizzes, and the system was sufficiently robust to function smoothly with the onslaught of students rushing to complete their work just before the deadline.
As with any voluntary survey, there were some limitations to the student survey that was conducted for this study.The timing of the survey may have affected the participation rate somewhat.As mentioned earlier, the survey was conducted during tutorials in the last week of the semester.A number of students indicated to lecturing staff that they chose not to attend tutorials during this last week as they preferred to spend the time finalising assessment tasks for other units.Furthermore, the students who do attend tutorials during the last week of the semester could be said to be the more conscientious, organised and structured in their approach to their learning.The timing of the survey may therefore have created some bias in the outcome.

Figure 1 .
Figure 1.A typical multi-part quiz question , for which the student has chosen to check the answer before continuing

Figure 4 .
Figure 4.A graph of quiz activity.No prizes for guessing the due date of this quiz!

Table 1 .
Participation rates for each quiz in STAT279, in which 493 students were enrolled

Table 2 .
Participation rates for each quiz in STAT379, in which 160 students were enrolled

Table 3 .
Percentage of students that agree and disagree with the following statements The MacQTeX quizzes are available for public access online at http://www.mq.edu.au/quizzes.Any quiz is accessible by entering guest as the Student Id.Acrobat Reader 5.0 or later is required.An installer for Mac OS X is also available for download on: http://sommelier.ics.mq.edu.au/MacQTeX/about/downloads.html