The Impact of Supplementary On-Line Resources on Academic Performance : A Study of First-Year University Students Studying Economics

This paper examines the impact of supplementary web-based materials on students’ academic performance in a first-year economics unit at university. In particular, the paper considers the impact of students’ usage of the unit’s webpage, voluntary on-line discussion board, voluntary on-line quizzes and voluntary on-line homework questions on their final mark in the first-year economics unit. It finds that students who make use of on-line materials perform better in the unit than students who do not, holding prior academic ability constant. The paper also finds that students’ usage of on-line supplementary materials does not vary according to their academic ability.


Introduction
In recent years the delivery of university courses has shifted from pure classroom-based instruction to the use of web-based resources.Such resources include lecture recordings, on-line discussion boards, on-line quizzes and pure web-based instruction.The use of web-based resources for students has several potential advantages, including: (i) it provides students with additional learning materials which they are able to revise; (ii) it provides students with the opportunity to share their knowledge; (iii) it allows students to catch up on course material if they miss a face-to-face lecture and, (iv) it provides students with flexibility on when they learn (see Brown and Ford, 2002 for further discussion on the potential advantages of the use of web-based teaching resources).
There are only a few studies in the education literature which consider the impact of the use of supplementary web-based materials on academic performance.For the most part, it has been reported that the use of supplementary on-line materials has a positive impact on students' marks at university (e.g., see Williams, Birch & Hancock, 2012;Woo et al., 2008;Gerber, Grundt & Grote, 2007;Mallik, 2011;Morris & Walker, 2006;Gosper et al., 2007 andCatley, 2004).For example, the findings by Mallik (2011) indicate that students' marks in a university unit increased by approximately 0.15 percent for every time they logged into the unit's webpage.Morris and Walker (2006) suggest that the introduction of supplementary web-based material increased the pass rates for a university unit by 20 percent.Cately (2004) reports that the average mark of students who participated in voluntary on-line quizzes was around 10 percentage points higher than students who do not participate in the on-line quizzes.
There are several limitations of this existing body of literature.First, most studies use small samples based on surveys of students and as a result are subject to selection bias issues.Second, most studies in the literature do not take into account students' prior academic ability when examining the relationship between academic performance and on-line university materials (e.g., Day &Foley, 2006, andCatley, 2004).It is possible that students' usage of on-line unit materials differs according to their prior academic ability.There is a wide body of literature that reports that students' prior academic ability, such as their scores on the university entrance examinations is a main factor influencing their performance at university (see Birch & Miller, 2004 for a review).Finally, the existing research mainly focuses on how students' usage of the unit's webpage impact on academic performance.For instance, Mallik (2011) only considers the number of times a student logs into the unit's webpage.Little is known on whether the impact of the use of on-line resources on students' academic performance varies across the types of on-line resources (such as quizzes and discussion boards).
The purpose of this paper is to enhance the understanding of the impact of web-based material on students' academic performance at university.Using Student Record File data from students enrolled in a principal economics unit, the paper estimates the impact of the usage of web-based materials on students' final mark in the unit of study, taking other factors, such as prior academic ability, into account.In particular, the paper looks at the impact of the students' usage of voluntary on-line quizzes, voluntary on-line homework questions, a voluntary on-line discussion board and the unit's webpage on their final mark for the unit.
The paper is structured as follows.Section 2 discusses the data, methodology and estimating procedure.Section 3 presents the empirical results and a summary and conclusion is presented in Section 4.

Method, Data and Estimating Procedure
The data for the analysis are drawn from the Student Records Files at the University of Western Australia.A benefit of the data set is that it contains information on all students enrolled at the university, rather than information gathered from a survey.The data set focuses on students enrolled in a first-year microeconomics unit in 2010.This unit offers a number of supplementary web-based materials for its students.These include a voluntary on-line discussion board, voluntary on-line quizzes and voluntary on-line homework questions assessable from textbook's supplementary materials website.The unit also has a webpage containing all unit materials and links to all the units' supplementary web-based materials.Data on students' usage of these on-line materials is captured through data collected in the university's Learning Management System (WebCT).The Student Record File data also contains information on students' prior academic abilities such as their score on the university entrance examinations (measured by their Tertiary Entrance Rank, TER), their marks for the unit, personal and demographic characteristics, such as gender and birthplace, and characteristics of the secondary school attended.The data sample is restricted to students with information on these characteristics.It is comprised of 1,012 students.Whether the student accessed the on-line homework questions for the unit. (a)   17.89% (181 students) NA NA (a) The data set only contains information on whether the student accessed the voluntary on-line homework materials.Information on the number of times a student accessed these materials was not available.
As shown in Table 1, all students in the sample accessed the unit's webpage on at least one occasion throughout the semester.The average number of times a student accessed the webpage was 53 times which is equivalent to approximately 4 times a week for the semester.Around 15 percent of students posted a comment on the voluntary discussion board.Of these students, the average number of posts was four.Almost all students (96 percent) attempted at least one of the voluntary on-line quizzes for the unit.Students who attempted the quizzes, on average, attempted 10 quizzes throughout the semester, equivalent to one quiz per topic for the course.Finally, around 18 percent of students attempted the voluntary on-line homework questions for the unit.
Table 2 shows that there are few differences in the students' use of on-line supplementary unit materials by their prior academic ability (based on their score on the university entrance examinations, TER).Hence, the mean TER score of students only varies by approximately 2 percentage points according to their usage of the units' webpage, comments on the discussion board and whether they accessed the on-line voluntary homework questions.As such, it is possible to suggest that for the most part, students' usage of supplementary on-line material is independent of their prior academic ability.  a) Students' TER score only varies significantly across their attempts at the voluntary on-line quizzes.It does not vary significantly for the other variables associated with on-line resources.
The model to estimate the impact of the use of supplementary web-based materials on academic performance is similar to that used in recent studies, such as Birch and Miller (2004).It is based on the education production function, whereby a student's academic performance (AP i ) can be expressed as a function of their prior educational attainment (E i ), ability (A i ) and their learning practises (LP i ).The model can be expressed as: Students' academic performance is measured by their final mark in the first-year economics unit (a mark out of one hundred).In the model, students' prior educational attainment is measured by their score on the university entrance examinations and whether they studied economics in high school.Students' ability is proxied by their gender and birthplace.Students' learning practices are measured by the type of high school attended, whether they are repeating the unit, their course major, whether they study part-time, if they are undertaking a double degree and their usage of the web-based materials for the unit.A variable controlling for the semester of study is also included in the model to capture any differences in students' marks across semesters.Following Birch and Miller (2004), the determinants of academic performance is estimated using OLS.This will allow for the impact of students' usage of supplementary material on final marks to be examined holding students' score on the university entrance examinations constant.Due to collinearity among the web-based materials variables, the impact of these variables on academic performance is estimated separately.A description of the variables used in the analyses is presented Table 3.

Results
The results from the model to estimate the determinants of academic performance are presented in Table 4. Panel (i) contains results from a model on academic performance which excludes variables on the use of supplementary web-based materials.Panel (ii) presents the results from the model controlling for the number of times that the student accessed the unit's webpage.Panel (iii) presents results from the model with variables on the number of times that a student posted a discussion on the voluntary on-line discussion board.Panel (iv) contains the results controlling for the number of voluntary on-line quizzes attempted and Panel (v) presents the results for the model controlling for the use of the voluntary on-line homework questions.Each panel contains the estimated coefficients and the absolute heteroscedastic corrected 't' statistics.The dependent variable is students' final mark in the first-year economics unit.On-Line Quizzes    (a) Each panel contains the estimated coefficients and the absolute 't' statistics in parenthesis.The symbol *** represents statistical significance at the 1 percent level, ** at the 5 percent level and * at the 10 percent level. (b) Not included in the estimating equation.
Many findings of the models to estimate the determinants of academic performance are similar to those presented in the existing literature (see Birch & Miller, 2004 for a review of studies).Hence, Panel (i) of Table 3 shows that students' final mark in first-year economics is positively correlated with their score on the university entrance examinations (TER), with a one percent increase in students TER score resulting in approximately a 0.8 percent increase in their final mark.Table 3 also shows that students' final mark in first-year economics is positively correlated with being born overseas, (OS), where overseas-born students have a mark advantage of approximately 3 percent compared to Australian-born students.
Panel (i) shows that students who attended a Catholic (Catholic) or Independent (Indp) high school, those repeating the unit (Repeat), those studying a degree other than commerce or economics (NoComm) and those who were studying a double degree (Double) have lower marks in first-year economics.Students from Catholic or Independent high schools have final marks which are 4.9 percent and 6.7 percent lower than the marks for students from Government schools.Students who were repeating the unit have marks that are 4.6 percent lower than the marks of non-repeaters.Studying a degree outside of commerce or economics is associated with approximately a 2.4 percent reduction in marks and studying a double degree is associated with approximately a 3 percent reduction in marks.Panels (ii) to (v) show that the inclusion of students' usage of web-based materials in the model of the determinants of students' academic success improves the explanatory power of the model.Hence, the adjusted R 2 increases from approximately 0.23 to around 0.30.This implies that variables on web-based materials substantially assist in the prediction of academic success.Moreover, the panels show that students' who make use of the on-line materials have higher marks in the unit than those who do not, holding their score on the university entrance examinations constant.Hence, for students with the same level of prior academic ability (measured by their score on the university entrance examinations) those who use supplementary web-based materials have higher marks than those who do not.Therefore, it appears that students' usage of supplementary web-based materials is a good indicator of their success in the first-year economics unit.Panel (ii) shows that there is positive correlation between students' final mark in first-year economics and the number of times they access the unit's web-page (Sessions).A five percent increase in the number of times a student accesses the unit's web-page is associated with a 1 percent increase in marks.The study by Mallick (2011) reports a similar finding.Panel (iii) shows that students' usage of the voluntary on-line discussion board also has a positive impact on their marks.Students who posted one discussion on the discussion board (DBP1) have marks which are 5.2 percent higher than the marks of students who did not post any discussions on the discussion board.Students who two discussions (DBP2) were found to have marks which are 8.6 percent higher than their counterparts not posting any discussions and students who posted at least three discussions on the discussion board have marks which are 9.3 percent higher than the marks of students without discussion board posts.
Students' use of the voluntary on-line quizzes is also a good indicator of their success in the first-year economics unit.Hence, Panel (iii) shows that students who attempted at least 11 quizzes have higher marks in the unit than students who did not attempt the quizzes.Students who attempted between 11 and 15 quizzes (Quiz11-15) had a mark advantage of 8.0 percent over students not attempting the quizzes.The mark advantage associated with attempting 16 to 20 quizzes (Quiz16-20) and 21 or more quizzes (Quiz>20) is 12.8 and 11.7 percent respectively.
Finally, Panel (v) shows that students who attempted the voluntary on-line homework questions (Homework) also have higher marks in the first-year economics unit than students who did not attempt the homework questions.These students' marks are approximately 5.5 percent higher than the marks of students who did not use the voluntary on-line homework questions.
There are three key findings from the analysis: (i) there is a positive impact on the usage of supplementary web-based materials on final marks.This impact is greater for students who access the materials more frequently throughout the semester.(ii) The impact of the usage of supplementary web-based material does not vary considerably over the type of web-based material access and (iii) the impact of the use of web-based materials on marks is considerably large compared to other determinants of academic success such as school type.

Summary and Conclusion
This paper has estimated the impact of students' usages of supplementary on-line materials on their final marks in a first-year economics unit at university.It has found that there is a positive correlation between students' usage of the on-line materials and their mark in the unit.The findings relating to the use of on-line materials on academic performance are quite pronounced.The results show that the impact of the use of on-line materials on academic performance is relatively large compared to other determinants of academic success.In addition, the results show that the impact of the use of on-line web-based materials on academic performance does not vary considerably across the type of web-based material.Students who use a larger number of the materials have the largest mark advantage.
The paper has also found that there is little difference in students' use of supplementary on-line web material according to their prior academic ability.As such, it must be other factors which motivate students to engage in the use of supplementary material.
The findings imply that there is possible merit to using supplementary web-based materials to enhance students' learning.Whether the findings hold for other subjects studied at university needs to be explored.It is also possible that the findings show that students who use the on-line resources are more motivated to study and therefore achieve higher marks in the university course.Further research is needed to distinguish between the 'pure' benefit to students of online material, in terms of helping their understanding of the course material overall, versus the fact that students who use online material are putting in more effort to study, and it is the extra effort that is leading to the higher marks, not the online material itself.More research is required to identify the factors which influence students' decisions to use on-line resources.Nevertheless, the findings in this paper show that students who make use of supplementary web-based materials perform better at university.As such the use of web-based materials appears to be a good indicator of students who will be successful in their university courses.

Table 1 .
Students' Usage of Web-Based Supplementary Material Over the Semester(a)

Table 2 .
Students' Usage of Web-Based Supplementary Material Over the Semester by Prior Academic Ability(a)

Table 3 .
Description of Variables Used in the Models of the Determinants of Academic Performance

Table 3 .
Description of Variables Used in the Models of the Determinants of Academic Performance Continued

Table 4 .
Description of Variables Used in the Models of the Determinants of Academic Performance(a)

Table 5 .
Description of Variables Used in the Models of the Determinants of Academic Performance Continued