Study of the Impacts of the Quality Assessment of Undergraduate Education Policy in China : Students ’ Perceptions

This paper analyzes a higher education policy issued in China in 2002: the Quality Assessment of Undergraduate Education Policy. It examines students’ perceptions of the policy impacts and students’ roles in the evaluation process by semi-structured interviews and questionnaire surveys. It reveals that the quality assessment in China has facilitated the improvement of teaching infrastructure and led to the intensification of school discipline in evaluated higher education institutions significantly. However, its impacts on teaching/learning are not noticeable. As an information publisher, quality assessment seems not very influential for students. Moreover, students do not think their voices have been demonstrated in the current quality assessment sufficiently, and expect their roles to be enhanced. Based on students’ perceptions of the policy impacts, this study ends up with an examination of the problems with quality assessment approaches and related suggestions to improve them.


Introduction
Quality of higher education has gained prominence in China over the last decade.In 2002, the Chinese Ministry of Education (MOE) issued the Project of Quality Assessment of Undergraduate Education, on the basis of some earlier informal evaluation regulations.Higher Education Evaluation Centre of the Ministry of Education (HEEC) was also established to undertake this specific assessment.In this project, all higher education institutions should be evaluated within a period of five years on a rolling basis.The quality assessment of undergraduate education focuses on teaching quality on the institutional level.The evaluation procedures are standardized and include self-assessments, site visits of external evaluators and follow-up reforms.The evaluation criteria were set up by the MOE, containing eight major indicators and 19 sub-indicators, used for all of the evaluated institutions.The first round of evaluation was finished at the end of June 2008, with 589 higher education institutions evaluated (HEEC, 2013).
The intended impact of the Quality Assessment of Undergraduate Education can be summarized based on the purposes defined in the policy discourse and the context in which the policy emerged.Firstly, the quality assemment intended to push the evaluated institutions to improve education quality by means of specifying their guiding principles on university operation, improving teaching facilities and university infrastructure, strengthening teaching management and facilitating the reforms of teaching and learning.Secondly, from the perspective of the state, it intended to reinforce the governance over higher education and push the evaluated institutions to do what the state wants them to do.Thirdly, the quality assessment scheme is also used as an information publisher, to promote the transparency and comparability of higher education institutions.
After working for one full cycle, the quality assessment policy caused lots of debates upon whether this project has realized its intended purposes.In this context, academic research is necessary to shed light on the policy consequences and impacts.The study of higher education quality assessment in China is still at the initial stage, and is dominated by the introduction of quality assessment and assurance practices in western countries.As for the Project of Quality Assessment of Undergraduate Education, much of the literature discussing it appears to lack supporting evidence from field investigation, like the research in other fields of higher education studies in China (Fan & Gao, 2010).Advocacy based on researchers' observation is frequently a substitute for in-depth analysis.Moreover, a great many recommendations have been made for improving the quality assessmewnt scheme.Unfortunately, these often originate from international experience but are not rooted in the problems of the Quality Assessment of Undergraduate Education per se.Consequently, these suggestions frequently ignore the context of Chinese higher education, and thus, lack feasibility.Thus, it is necessary to conduct empirical studies on the impact of the quality assessment scheme from the perspectives of different stakeholders.This paper focuses on students' perceptions of the impacts of the quality assessment.As mentioned above, one of the fundamental objectives of the quality assessment policy is to improve the quality of higher education provision for students and to inform students (and other stakeholders) of the real quality of higher education institutions; students are the primary beneficiaries of the quality assessment (MOE, 2002).Thus, students' voices are essential to evaluate the policy impacts.Furthermore, students, who have been gradually regarded as the "consumers" of universities in the context of massification and commercialization of higher education, are fully entitled to be involved in both defining and evaluating quality, as suggested by Hill (1995).Thus, it is significant to examine the policy efficiency from the perspective of students, as well as to explore their participation in the quality assessment process.This paper will show the findings from the empirical studies.It begins with a description of impacts of the quality assessment on improving the quality provision in different levels of higher education institutions together with the continuity of these impacts.The effects of the quality assessment on increasing the transparency of higher education quality will also be examined.Besides the positive outcomes of the quality assessment, the paper will also reveal its negative effects on the evaluated institutions and students briefly.In addition, students' involvement in the quality assessment process will be discussed, including their approaches of contributing to the quality assessment, their satisfaction on the current participation and their role expectations.

Method
In this research, the main research methods are semi-structured interviews and questionnaire surveys.Qualitative interviews have been carried out at first to inform the development of a questionnaire.The sample frame is the undergraduate students of 589 higher education institutions which were evaluated in the first round of evaluation.These students have experienced the three stages: pre-evaluation, evaluation, and post-evaluation in their universities.Thus, it is supposed that they can perceive the policy impacts and problems at first hand.A purposive sampling was used to select the students being interviewed and surveyed, on the basis of knowledge of the population and the purposes of the study.According to the initial hypotheses deriving from the literature review, the impacts of the quality assessment policy on various levels of institutions might be different (Brennan & Shah, 2002;Schwarz & Westerheijden, 2004).Thus, the sample of students in the institutions with various educational levels was intended to be selected.However, there is not an acknowledged and reliable reference of universities' levels in China currently.I had to use reputation as a basic reference to choose the institutions with different levels of educational quality.In China, universities that are governed by Ministry of Education directly are regarded as being on the elite level of the reputation hierarchy as a thin layer.Those institutions funded and administrated by local authorities can be categorized into top, high, middle and low levels in their provinces (MOE, 2006).The sample in this research involves all of the different levels of higher education institutions.
The research design was developed on the basis of both qualitative and quantitative methods.Initially, five students from different-level universities were selected; face-to-face interviews were conducted with them in Chinese.These interviews averaged 45 minutes and were audio-recorded; they were then transcribed and translated into English.Based on the analysis of the qualitative data, subsequently, a questionnaire was designed, which comprises 16 questions, covering:  Students' perceptions of the impacts of the quality assessment on the improvement of teaching infrastructure, teaching management, and the process of teaching/learning;  The impacts of the quality assessment on information publication;  The negative effects of the quality assessment on the evaluated universities and students;  Students' involvement in the process of the quality assessment.
These variables were developed from the policy objectives defined in the policy document and the performance indictors used in the quality assessment (Note 1).This research measured the dimensions which are related with students.The questionnaires were delivered to 636 students in seven universities with different statuses and 610 valid ones were returned: a response rate of 95.91%.The characterizations of the higher education institutions selected in the sample are shown in Table 1.Concerning the survey, the data collected was analyzed through descriptive statistics.Students were asked to label the impacts of higher education quality assessment on the improvement of teaching quality concerning teaching infrastructure, teaching management and the process of teaching/learning per se, on a scale of "improved a lot", "improved somewhat", "stayed the same", "got worse" and "can not say".As displayed in Table 2, firstly, the quality assessment has a strong impact on the improvement of teaching infrastructure.72.0% of the students surveyed think the facilities in the evaluated institutions have been improved.When it comes to teaching management, students feel that the quality assessment has a substantial effect on "student discipline" (Note 2).63.0% of the respondents feel that the quality assessment has made the discipline stricter.As for teaching/learning, the statistical analysis indicates that the impact of quality assessment on the teaching methods is not very noticeable.About two third of the answerers think the teaching methods have not been changed at all as a result of the quality assessment.Compared with the teaching methods, the quality assessment has contributed more to teachers' commitment.44.6% of the answerers believe that the quality assessment has provided impetus to increasing teachers' commitment.Similarly, to some extent, the quality assessment has fairly encouraged students' commitment to study.About half of the respondents contended that they have devoted more to their study since the quality assessment.In addition, to some extent, the quality assessment has a positive effect on student performance assessment in the evaluated universities.In this sample, 43.2% perceive that the approaches to assessing student performance have become better because of the external evaluation.In sum, it appears that the impacts of the quality assessment on the different aspects of teaching quality provision are not equivalent.Students' perceptions indicate that, the external quality assessment has considerably facilitated the improvement of teaching infrastructure and teaching management (student discipline).However, the outcomes regarding teaching/learning are scarce.To some extent, the quality assessment has contributed to the reforms of student performance assessment as well as encouraged teachers' and students' commitments, while the impact of the quality assessment on the innovations of teaching methods is trivial.
In fact, the impacts of the quality assessment on the various aspects have interplayed and integrated.The semi-structured interviews reveal, for example, that the improvement of students' commitment to study has not been reached directly by the quality assessment.It is the improvement of teaching facilities, teaching methods, teachers' commitment to classes and the teaching management (mainly means student discipline here) that impel students' devotion to study.The possibility that the quality assessment can realize its objective-improvement depends on the integration of all of these positive consequences.

The Continuity of the Impacts of the Quality Assessment
In this research, the continuity of the quality assessment's impacts was studied, using the dimension of student discipline as a case.Figure 1 presents a rebound curve, proposed by four respondents of the semi-structured interviews.With the implementation of the quality assessment policy, the student discipline had become much stricter than that before the evaluation (per-evaluation).However, since the external evaluators left, the stricter discipline in the evaluated institutions has regressed to some extent.Thus, the discipline in the post-evaluation is less strict than that in the process of evaluation, but stricter than that in pre-evaluation.The survey draws the similar conclusions: about two third the respondents surveyed (61.2%) think the stricter discipline has regressed somewhat after the quality assessment; while only 8.2% of them feel it has gone back a lot and 17.8% think it has been kept completely.In particular, the semi-structured interviews show us that the extent of the regression was related with the administrative culture of the evaluated universities.In the institutions with strong autonomy, the bureaucratic management came abruptly as a result of the quality assessment.After the external evaluators left, it would be difficult for them to keep these changes without an appropriate context.In contrast, in the universities without much autonomy, the stricter disciplines did not conflict with the normal administration as strongly as in the autonomous institutions.Thus, these changes could be kept easily.

The Impacts of the Quality Assessment on Different Levels of Higher Education Institutions
When it comes to the relationship between the degree of the policy impacts and the level of the evaluated institutions, at first, I categorized the universities referring to their positions in the reputation hierarchy as indicated in Table 1, i.e., elite universities governed by the Ministry of Education as 1, top universities in local areas as 2, high-level institutions as 3, middle-level ones as 4, and low-level institutions as 5.That is to say, the lower the number, the higher the university's reputation.Subsequently, in each group, taking "can't say" as missing data and scoring "improved a lot" as 1, "improved somewhat" as 2, "stayed the same" as 3, and "got worse" as 4, a mean score of students' perceptions of each variable was calculated.Thus, the lower the mean score, the more significant the quality improvement.
As presented in Table 3, there is a gradual change between the five groups of higher education institutions with different levels in the extent to which the quality assessment has influenced their education quality.By and large, as we move from the elite institutions to the inferior ones, from Group 1 to Group 5, the impacts are more and more significant.Notwithstanding, the impacts of quality assessment on the lowest-level institutions (Group 5) are not very considerable.Its effects on all of the variables discussed, except for the student discipline, are not as high as those in the middle-level institutions (Group 4).In order to measure the realization of the policy objective: information publication, i.e., whether the quality assessment has provided effective information for students, a question was designed by letting students compare the quality assessment with its main competitor: university rankings.The survey reveals that 43.5% of the answerers think university rankings are more useful, while only 11.2% prefer the information published by the quality assessment.
The respondents in the semi-structured interviews indicated that the quality assessment is less helpful than the university rankings, because (1) the quality assessment is full of falsification and its result is considerable distortion; (2) the league-table seems clear at a glance.On the opposite, the four-grade scale (excellent, good, qualified and unqualified) used by the quality assessment is not very effective to distinguish the quality of higher education institutions; especially, almost every evaluated university is excellent/good (Note 3); (3) university league tables, using more avenues of mass media to publish the information, are more influential than quality assessment, etc.In contrast, other students contended that the quality assessment is more reliable to make the quality of the evaluated institutions transparent, because it depends on the on-site visits of several experienced experts rather than the rigid statistics, which makes its evaluation results more convincing.

The Negative Effects of the Quality Assessment
As indicated by all of the interviewees, quality assessment has not only facilitated the development of the evaluated universities, but also brought on some negative effects.Of all the 343 answers to the open question in the survey, more than 170 students think that the quality assessment policy has caused the falsification of materials and disturbed the routine work of the evaluated higher education institutions, and about 200 students criticize that the quality assessment has given lots of trouble to them.The frequencies are shown in Table 4. Based on the responses of the interviewed students, an unintended consequence of the quality assessment was detected, i.e., the evaluated universities were inclined to train their students to get ready for the visits of the external evaluators.This training involved the basic knowledge of the quality assessment (e.g., who are the beneficiaries of the quality assessment?), the information of the university (e.g., the history of the university, the faculties and departments it has, and its advantages), and some basic skills for students (such as English and computer skills).After the training courses, students even need to pass related examinations.
Most of the universities have organized the pre-training for students, except for the ones with top status probably (University A and University F in this sample).This means the awareness that the quality assessment is going to happen has not induced the same responses from all of the universities, which might be related with their education levels.After the pre-training, most of students think they became more knowledgeable of their own institutions ( 261); and about half of them improved their basic skills (195).At the same time, some students also complain that the training led to excessive bother (117).That is to say, the effects of pre-training are two-sided, as shown in Table 5.
Table 5. Students' benefits and disbenefits from the pre-training

Students' benefits and disbenefits Frequency
Got more knowledge of their own universities 261 Improved their basic skills (e.g., English and computer skills) 195 No benefits 28 Disbenefits (much trouble) 117 Others 7 No. of respondents 388

Students' Participation and Their Role Expectations
The main approaches of students to participate in the quality assessment involved helping to prepare for external evaluators' visits, being the interviewees of evaluators and the subjects of observations (e.g., in-class inspection).
As for students' opinions, among all of the 18 respondents who were interveiwed by external evaluators, no one think their ideas have always been taken into account by the expert committee, 10 of them said they have been considered contingently, while 5 of them said that their voices had never been heard.
Regarding the participation of students in the quality assessment, it is suggested that the representatives of students should be included in the evaluator committee (Liu & Rosa, 2008).In this research, I use whether students think it is necessary to add student representatives in the expert panel as an indicator to test students' role expectations in the quality assessment.The survey results illustrate that students believing the evaluator committee needs a student representative (69.2%) are much more than those with opposite ideas (7.5%).The percentage of the respondents that remain ambiguous is also relatively high: 23.4%.
According to the responses of the five interviewees along with 362 answerers to the open question in the surveys, the reasons why they argue for or against student representatives to be taken in the expert committee can be summarized as follows.On the one hand, students who regard it as necessary contend that:  As an important stakeholder of higher education and the primary beneficiary of the quality assessment, students have the right to take part in the quality assessment and demonstrate their own voices;  Students have a specific perspective of quality in higher education, which is different from teachers' and other stakeholders' conceptions.The participation of student representatives will contribute to the quality assessment and a possible improvement from a new perspective.Furthermore, student representatives have their own advantages as members of the expert committee.For example, the external evaluators have not been in an active dialogue with students in the current quality assessment process.Compared with these professors and/or university leaders (Note 4), it seems easier for student representatives to communicate with their peers and gain the information about the real quality of the evaluated universities from the perspective of students.In addition, the attendance of students in the evaluator committee might be helpful to decrease the corruption in the evaluation processes and increase the fairness and accountability of the evaluation results;


The join of student representatives in the expert panel will probably become a starting point of the democratization of university administration with students' involvement.
On the other hand, some students are pessimistic about the participation of the student representatives, because:


It is difficult for them to make a voice when working with the high-level experts.The words of the lowly person always carry little weight;  Some students also suspect the capabilities of student representatives and do not believe they would be qualified to be quality evaluators.

Discussion
According to students' perceptions, first of all, it appears that the quality assessment of undergraduate education has contributed considerably to the improvement of teaching facilities and management (student discipline).On the contrary, its effects on teaching/learning, especially the innovations of teaching methods, are not very strong.Thus, the quality assessment does not have the equivalent impacts on different dimensions of the quality provision of universities.Moreover, to some extent, the effects of the quality assessment regressed after the external evaluators left.
Probing into reasons, on the one hand, the limitations of the quality assessment per se should be taken into account; the potential for the external quality assessment scheme to contribute to the long-term teaching/learning improvement should not be overestimated.As indicated by Vroeijenstijn (1995), there might never be a direct link between the activities in the field of quality assessment and improvement.Thus, we could not rely on the quality assessment system excessively to improve the quality provision, or even to solve all of the existing problems in higher education.
On the other hand, the quality assessment policy implemented in China should be reformed to increase its efficiency on teaching/learning.In this regard, I suggest that the internal quality assurance schemes should be emphasized (Note 5), which is just one of the 19 sub-indicators in the existing evaluation criteria.First of all, it is found that the internal reviews and assessments are more valid and fruitful for quality improvement than those done by outside evaluators (Harvey & Newton, 2007).In addition, as mentioned before, there is interplay between the original conceptions of university running and the intruded requirements of the external quality assessment.When they match very well, the effects of the quality assessment will be easily kept and might become long-term impacts; on the opposite, when they conflict a lot, the effects will regress soon.Thus, the impacts on teaching/learning improvement would become continuing, if we had linked internal and external quality evaluation schemes successfully.It is suggested that the design and operation of serious and tough internal reviews of quality can be monitored through the external evaluators, who will become their consultants (Note 6) of quality assurance rather than only "box tickers".With the link of internal and external reviews, the improvement of teaching/leaning might be stronger and last longer.
Secondly, the impacts of the quality assessment on higher education institutions with different levels are not equivalent.On the whole, as we move from the top institutions to the inferior ones, the effects are more and more significant.In the context that using the same set of performance indicators to evaluate all of the higher education institutions in China, compared with the elite institutions, there are more space and necessity for the second-rate ones to improve their teaching quality.At the same time, in the process of getting ready for the external assessment visits, the responses of various levels of institutions are also different.The institutions with low statuses have devoted more (such as organizing pre-training courses for students) to the "window dressing" than the top ones.This could be explained by the finding that institutions with an elite reputation have more autonomy in China and thus need not to follow the government's expectations quickly and compliantly (Csizmadia, Enders, & Westerheijden, 2008;Yang, Vidovich, & Currie, 2007).
However, as indicated before, the institution at the lowest level is exceptional, on which the impacts of the quality assessment are not as high as those at the middle level.The limited resources available in the most inferior institutions probably make it difficult to improve their teaching infrastructure and to employ qualified teachers.In addition, to some extent, the impacts of the quality assessment on the improvement of education quality depend upon the commitment of the evaluated institutions' leaders (Csizmadia et al., 2008).The leaders of the inferior institutions might not have much ambition and capabilities to initiate some reforms.In particular, no institutions have been regarded as unqualified until now.Due to lack of efficient mechanisms to exert certain punishment on the institutions with really poor performance, some institutions on the bottom of the "pyramid" might have no motivation to move up.
Thus, the standardized evaluation criteria used in the current quality assessment are suggested to be replaced by the diversified ones aiming at different levels of institutions.Elite universities might be exempted from the quality assessment.At the same time, the ones at the bottom should be imposed on more pressure to improve their quality through equipping them with accreditation procedures or punishment mechanisms.Thus, the institutions with "really poor teaching" would be weeded out.
Thirdly, it is found that as an information publisher, the quality assessment is less popular with students than its competitor: university rankings.The four-grade scale and the result that excellent/good institutions seem out of proportion have lowered the usefulness of the information published by the quality assessment.In this case, does the quality assessment become a way to facilitate the quality improvement or a strategy to show accountability and even to conceal the essential quality decline?These doubts might deserve further reflection.In addition, as discussed before, the implementation of the quality assessment policy has also brought lots of negative effects to the evaluated universities and students, such as the falsification of evaluation documents.That means there is still much room for the quality assessment to improve its efficiency, as suggested by Liu and Rosa (2008).
With regard to the roles of students in the quality assessment system, students participate mainly through taking the pre-training courses and helping the rehearsals towards a forthcoming site visit.However, most of the surveyed students do not think they have been involved in the quality assessment sufficiently.The external evaluators have not been in an active dialogue with students to understand students' perceptions of the quality of higher education or their concepts of "quality".However, students have their rights and advantages to demonstrate their voices and thus they expect that their roles could be enhanced in the process of quality assessment.In this context, I suggest that the Quality Assessment of Undergraduate Education Policy should pay more attention to students' definitions of "quality" when its performance indicators are designed.During the site-visits, external evaluations are suggested to spend more time on the communication with students rather than only discussing with the institution leaders.At the same time, student representatives are expected to be included in the evaluator committee, which might not only add a new perspective to examine the quality, but also become a promising initiation of the democratization of higher education institutions with the involvement of students.
Linking the initial semi-structured interviews and subsequent questionnaire surveys, this research examines the impacts of the quality assessment policy in China from the perspective of students.However, because of the limitation of students' knowledge, the measure of the policy impacts in the survey is not all-around.For example, the effects on teaching management only involve the student discipline, while the changes of the administrative organizations and the establishment of teaching regulations, which are far away from students' lives, have not been touched.Thus, this research has provided the students' perceptions of the impacts of the quality assessment, but is far from presenting a whole picture, which needs to be explored in future research.

Notes
Note 1.The evaluation criteria set out by MOE contains eight major performance indicators, including: (1) the direction of institutions' development, which means the orientation of institutions and the rationale of university running; (2) the staff, which includes staff numbers and their qualifications; (3) the facilities for teaching, i.e., the infrastructure and financing for teaching; (4) disciplines and teaching, including the discipline structures, curriculum (curriculum design, the usage of textbooks, the teaching methods and contents, and the assessment of students' performance, etc.) and practical training courses (such as internship and workshops); (5) teaching management, containing the administration and internal quality assurance; (6) the learning atmosphere, which represents teachers' commitment to teaching and students' commitment to study; (7) teaching outcomes, which include the knowledge, skills and morality of students, the quality of graduation projects and theses, physical training, employments and the reputation of institutions; (8) specific characteristics of each university (MOE, 2002).Note 3.According to the published evaluation results of the 502 higher education institutions evaluated from 2003 to 2007, the numbers of excellent, good, qualified and unqualified universities were respectively 353, 128, 21, and 0 (HEEC, 2008).That is to say, 95.82% of the institutions assessed were excellent/good, while no institutions were considered unqualified.
Note 4. The evaluators working for the Quality Assessment of Undergraduate Education Project are scholars of various disciplines with high academic reputations and/or management experience.They were nominated by universities and appointed by the Ministry of Education (HEEC, 2008).
Note 5.The semi-structured interviews show that the internal quality assurance schemes are still on the initial phase in Chinese higher education institutions.
Note 6.According to Csizmadia et al. (2008), the involvement of external consultants has strong effects on the implementation of quality management.

LooseFigure 1 .
Figure 1.The changes of student discipline

Note 2 .
Student discipline refers to a standard specified by the institution to help control student behavior by coordinating the university's disciplinary procedures and by informing students what types of behavior are expected of them and what types are forbidden.

Table 1 .
The higher education institutions in the sample The Impacts of the Quality Assessment on the Improvement of Teaching Quality

Table 2 .
The impacts of the quality assessment on the improvement of teaching quality

Table 3 .
The impact of the quality assessment on different levels of HEIs

Table 4 .
The negative effects of the quality assessment